2024-09-21T23:35:04.277 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T23:35:04.298 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T23:35:04.388 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914544 branch: squid description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/compare-mirror-images-krbd} email: ceph-qa@ceph.com first_in_suite: false job_id: '7914544' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each name: teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.4: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.5: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.6: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 extra_system_packages: - pv thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 1478 sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 sleep_before_teardown: 0 subset: 24/128 suite: rbd suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDhw5lLtglvs+U6XQ4fvivqzVMXu4Kh9MFA5QdoAONywSQ1nIxXy+vROrP/azpMzYiv2d39Yuw9pTD29gN4jbrI= smithi060.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNGh2LIWkdpywhXgv5VVuZObANPS13LaR+dESgkBDuxezy2skEuA6fzULv3le0MNVsP4tfY5ZsadoZw+oPrjpgs= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - workunit: clients: cluster1.client.mirror: - rbd/compare_mirror_images.sh env: RBD_DEVICE_TYPE: krbd timeout: 3h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_21:16:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T23:35:04.388 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa; will attempt to use it 2024-09-21T23:35:04.389 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks 2024-09-21T23:35:04.389 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T23:35:04.391 INFO:teuthology.task.internal:Checking packages... 2024-09-21T23:35:04.410 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '4e45e6b490e6cbe082cdf155fea780438ec53fc7' 2024-09-21T23:35:04.410 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T23:35:04.410 INFO:teuthology.packaging:ref: None 2024-09-21T23:35:04.410 INFO:teuthology.packaging:tag: None 2024-09-21T23:35:04.411 INFO:teuthology.packaging:branch: squid 2024-09-21T23:35:04.411 INFO:teuthology.packaging:sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T23:35:04.411 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-09-21T23:35:04.606 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-373-g4e45e6b4-1jammy 2024-09-21T23:35:04.607 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T23:35:04.614 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T23:35:04.614 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T23:35:04.716 INFO:teuthology.task.internal:Saving configuration 2024-09-21T23:35:04.734 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T23:35:04.859 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T23:35:04.883 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914544', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 23:29:32.760486', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIRJYK0LlNctF3BaZs3qziKVfrS+J+J9ff+0GWmfyJT2'} 2024-09-21T23:35:04.903 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914544', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 23:29:32.761975', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBtSTGjA5T5+LXU+DCeF4MKuGGlSKkwnm3fdmr1PKsNx'} 2024-09-21T23:35:04.903 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T23:35:04.966 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-21T23:35:04.966 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-21T23:35:04.967 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T23:35:05.210 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbae0397490>, signals=[15]) 2024-09-21T23:35:05.210 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T23:35:05.216 INFO:teuthology.task.internal:Opening connections... 2024-09-21T23:35:05.216 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2024-09-21T23:35:05.218 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T23:35:05.298 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:35:05.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T23:35:05.377 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T23:35:05.404 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2024-09-21T23:35:05.409 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2024-09-21T23:35:05.409 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2024-09-21T23:35:05.455 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-21T23:35:05.455 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="Ubuntu" 2024-09-21T23:35:05.455 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="22.04" 2024-09-21T23:35:05.455 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-21T23:35:05.455 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_CODENAME=jammy 2024-09-21T23:35:05.455 INFO:teuthology.orchestra.run.smithi031.stdout:ID=ubuntu 2024-09-21T23:35:05.455 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE=debian 2024-09-21T23:35:05.456 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-21T23:35:05.456 INFO:teuthology.orchestra.run.smithi031.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-21T23:35:05.456 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-21T23:35:05.456 INFO:teuthology.orchestra.run.smithi031.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-21T23:35:05.456 INFO:teuthology.orchestra.run.smithi031.stdout:UBUNTU_CODENAME=jammy 2024-09-21T23:35:05.456 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2024-09-21T23:35:05.478 DEBUG:teuthology.orchestra.run.smithi060:> uname -m 2024-09-21T23:35:05.482 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2024-09-21T23:35:05.482 DEBUG:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2024-09-21T23:35:05.527 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-21T23:35:05.527 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="Ubuntu" 2024-09-21T23:35:05.527 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="22.04" 2024-09-21T23:35:05.527 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-21T23:35:05.528 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_CODENAME=jammy 2024-09-21T23:35:05.528 INFO:teuthology.orchestra.run.smithi060.stdout:ID=ubuntu 2024-09-21T23:35:05.528 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE=debian 2024-09-21T23:35:05.528 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-21T23:35:05.528 INFO:teuthology.orchestra.run.smithi060.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-21T23:35:05.528 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-21T23:35:05.528 INFO:teuthology.orchestra.run.smithi060.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-21T23:35:05.528 INFO:teuthology.orchestra.run.smithi060.stdout:UBUNTU_CODENAME=jammy 2024-09-21T23:35:05.528 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2024-09-21T23:35:05.552 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T23:35:05.568 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T23:35:05.636 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T23:35:05.636 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2024-09-21T23:35:05.639 DEBUG:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2024-09-21T23:35:05.643 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T23:35:05.770 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T23:35:05.771 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2024-09-21T23:35:05.773 DEBUG:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2024-09-21T23:35:05.780 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T23:35:06.105 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T23:35:06.121 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T23:35:06.121 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T23:35:06.122 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T23:35:06.122 DEBUG:teuthology.orchestra.run.smithi060:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T23:35:06.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T23:35:06.127 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-09-21T23:35:06.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T23:35:06.129 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-09-21T23:35:06.176 INFO:teuthology.orchestra.run.smithi031.stdout:5.15.0-56-generic 2024-09-21T23:35:06.176 INFO:teuthology.task.kernel:Running kernel on smithi031: 5.15.0-56-generic 2024-09-21T23:35:06.176 DEBUG:teuthology.orchestra.run.smithi031:> sudo apt-get clean 2024-09-21T23:35:06.177 INFO:teuthology.orchestra.run.smithi060.stdout:5.15.0-56-generic 2024-09-21T23:35:06.177 INFO:teuthology.task.kernel:Running kernel on smithi060: 5.15.0-56-generic 2024-09-21T23:35:06.178 DEBUG:teuthology.orchestra.run.smithi060:> sudo apt-get clean 2024-09-21T23:35:06.310 DEBUG:teuthology.orchestra.run.smithi031:> sudo apt-get update 2024-09-21T23:35:06.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo apt-get update 2024-09-21T23:35:06.439 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T23:35:06.441 INFO:teuthology.orchestra.run.smithi060.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-21T23:35:06.441 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-21T23:35:06.472 INFO:teuthology.orchestra.run.smithi031.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-21T23:35:06.514 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-21T23:35:06.576 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T23:35:06.657 INFO:teuthology.orchestra.run.smithi031.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-21T23:35:06.680 INFO:teuthology.orchestra.run.smithi031.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,839 kB] 2024-09-21T23:35:06.683 INFO:teuthology.orchestra.run.smithi060.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,839 kB] 2024-09-21T23:35:06.782 INFO:teuthology.orchestra.run.smithi031.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [544 kB] 2024-09-21T23:35:06.786 INFO:teuthology.orchestra.run.smithi031.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [298 kB] 2024-09-21T23:35:06.788 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,062 kB] 2024-09-21T23:35:06.789 INFO:teuthology.orchestra.run.smithi031.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-21T23:35:06.789 INFO:teuthology.orchestra.run.smithi031.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,431 kB] 2024-09-21T23:35:06.809 INFO:teuthology.orchestra.run.smithi031.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-21T23:35:06.810 INFO:teuthology.orchestra.run.smithi031.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [418 kB] 2024-09-21T23:35:06.811 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [544 kB] 2024-09-21T23:35:06.811 INFO:teuthology.orchestra.run.smithi060.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [298 kB] 2024-09-21T23:35:06.813 INFO:teuthology.orchestra.run.smithi060.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-21T23:35:06.813 INFO:teuthology.orchestra.run.smithi060.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-21T23:35:06.814 INFO:teuthology.orchestra.run.smithi031.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-21T23:35:06.814 INFO:teuthology.orchestra.run.smithi031.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [905 kB] 2024-09-21T23:35:06.814 INFO:teuthology.orchestra.run.smithi060.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,431 kB] 2024-09-21T23:35:06.821 INFO:teuthology.orchestra.run.smithi031.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [629 kB] 2024-09-21T23:35:06.825 INFO:teuthology.orchestra.run.smithi031.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [177 kB] 2024-09-21T23:35:06.827 INFO:teuthology.orchestra.run.smithi031.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.3 kB] 2024-09-21T23:35:06.827 INFO:teuthology.orchestra.run.smithi031.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-21T23:35:06.827 INFO:teuthology.orchestra.run.smithi031.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-21T23:35:06.833 INFO:teuthology.orchestra.run.smithi060.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [418 kB] 2024-09-21T23:35:06.837 INFO:teuthology.orchestra.run.smithi060.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-21T23:35:06.837 INFO:teuthology.orchestra.run.smithi060.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [629 kB] 2024-09-21T23:35:06.841 INFO:teuthology.orchestra.run.smithi031.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-21T23:35:06.841 INFO:teuthology.orchestra.run.smithi031.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-21T23:35:06.841 INFO:teuthology.orchestra.run.smithi060.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [905 kB] 2024-09-21T23:35:06.842 INFO:teuthology.orchestra.run.smithi060.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [703 kB] 2024-09-21T23:35:06.848 INFO:teuthology.orchestra.run.smithi060.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [177 kB] 2024-09-21T23:35:06.848 INFO:teuthology.orchestra.run.smithi060.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [356 kB] 2024-09-21T23:35:06.849 INFO:teuthology.orchestra.run.smithi060.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.3 kB] 2024-09-21T23:35:06.887 INFO:teuthology.orchestra.run.smithi060.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-21T23:35:06.887 INFO:teuthology.orchestra.run.smithi060.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-21T23:35:06.887 INFO:teuthology.orchestra.run.smithi060.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-21T23:35:06.887 INFO:teuthology.orchestra.run.smithi060.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-21T23:35:06.887 INFO:teuthology.orchestra.run.smithi060.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-21T23:35:06.887 INFO:teuthology.orchestra.run.smithi060.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-21T23:35:06.888 INFO:teuthology.orchestra.run.smithi060.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,499 kB] 2024-09-21T23:35:06.904 INFO:teuthology.orchestra.run.smithi060.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [430 kB] 2024-09-21T23:35:06.908 INFO:teuthology.orchestra.run.smithi060.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-21T23:35:06.908 INFO:teuthology.orchestra.run.smithi060.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,125 kB] 2024-09-21T23:35:06.916 INFO:teuthology.orchestra.run.smithi060.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [733 kB] 2024-09-21T23:35:06.921 INFO:teuthology.orchestra.run.smithi060.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-21T23:35:06.923 INFO:teuthology.orchestra.run.smithi060.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-21T23:35:06.923 INFO:teuthology.orchestra.run.smithi060.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-21T23:35:06.925 INFO:teuthology.orchestra.run.smithi060.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-21T23:35:06.925 INFO:teuthology.orchestra.run.smithi060.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-21T23:35:06.939 INFO:teuthology.orchestra.run.smithi060.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-21T23:35:06.939 INFO:teuthology.orchestra.run.smithi060.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-21T23:35:06.940 INFO:teuthology.orchestra.run.smithi060.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-21T23:35:06.940 INFO:teuthology.orchestra.run.smithi060.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-21T23:35:06.956 INFO:teuthology.orchestra.run.smithi060.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-21T23:35:06.956 INFO:teuthology.orchestra.run.smithi060.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-21T23:35:06.956 INFO:teuthology.orchestra.run.smithi060.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-21T23:35:06.957 INFO:teuthology.orchestra.run.smithi060.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-21T23:35:06.957 INFO:teuthology.orchestra.run.smithi060.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-21T23:35:07.013 INFO:teuthology.orchestra.run.smithi031.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-21T23:35:07.152 INFO:teuthology.orchestra.run.smithi031.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [703 kB] 2024-09-21T23:35:07.347 INFO:teuthology.orchestra.run.smithi031.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,062 kB] 2024-09-21T23:35:07.507 INFO:teuthology.orchestra.run.smithi031.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [356 kB] 2024-09-21T23:35:07.521 INFO:teuthology.orchestra.run.smithi031.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-21T23:35:07.522 INFO:teuthology.orchestra.run.smithi031.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-21T23:35:07.523 INFO:teuthology.orchestra.run.smithi031.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,499 kB] 2024-09-21T23:35:07.592 INFO:teuthology.orchestra.run.smithi031.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [430 kB] 2024-09-21T23:35:07.609 INFO:teuthology.orchestra.run.smithi031.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-21T23:35:07.609 INFO:teuthology.orchestra.run.smithi031.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,125 kB] 2024-09-21T23:35:07.642 INFO:teuthology.orchestra.run.smithi031.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [733 kB] 2024-09-21T23:35:07.661 INFO:teuthology.orchestra.run.smithi031.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-21T23:35:07.666 INFO:teuthology.orchestra.run.smithi031.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-21T23:35:07.671 INFO:teuthology.orchestra.run.smithi031.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-21T23:35:07.672 INFO:teuthology.orchestra.run.smithi031.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-21T23:35:07.673 INFO:teuthology.orchestra.run.smithi031.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-21T23:35:07.749 INFO:teuthology.orchestra.run.smithi031.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-21T23:35:07.749 INFO:teuthology.orchestra.run.smithi031.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-21T23:35:07.750 INFO:teuthology.orchestra.run.smithi031.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-21T23:35:07.752 INFO:teuthology.orchestra.run.smithi031.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-21T23:35:07.752 INFO:teuthology.orchestra.run.smithi031.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-21T23:35:07.752 INFO:teuthology.orchestra.run.smithi031.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-21T23:35:07.753 INFO:teuthology.orchestra.run.smithi031.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-21T23:35:07.834 INFO:teuthology.orchestra.run.smithi031.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-21T23:35:07.834 INFO:teuthology.orchestra.run.smithi031.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-21T23:35:13.598 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 16.3 MB in 2s (7,108 kB/s) 2024-09-21T23:35:13.636 INFO:teuthology.orchestra.run.smithi031.stdout:Fetched 16.3 MB in 2s (7,205 kB/s) 2024-09-21T23:35:15.073 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2024-09-21T23:35:15.081 INFO:teuthology.orchestra.run.smithi031.stdout:Reading package lists... 2024-09-21T23:35:15.091 DEBUG:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T23:35:15.099 DEBUG:teuthology.orchestra.run.smithi031:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T23:35:15.151 INFO:teuthology.orchestra.run.smithi031.stdout:Reading package lists... 2024-09-21T23:35:15.158 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2024-09-21T23:35:15.342 INFO:teuthology.orchestra.run.smithi031.stdout:Building dependency tree... 2024-09-21T23:35:15.342 INFO:teuthology.orchestra.run.smithi031.stdout:Reading state information... 2024-09-21T23:35:15.376 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2024-09-21T23:35:15.377 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2024-09-21T23:35:15.515 INFO:teuthology.orchestra.run.smithi031.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T23:35:15.515 INFO:teuthology.orchestra.run.smithi031.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T23:35:15.515 INFO:teuthology.orchestra.run.smithi031.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T23:35:15.515 INFO:teuthology.orchestra.run.smithi031.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T23:35:15.515 INFO:teuthology.orchestra.run.smithi031.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-21T23:35:15.516 INFO:teuthology.orchestra.run.smithi031.stdout: python2.7-minimal 2024-09-21T23:35:15.516 INFO:teuthology.orchestra.run.smithi031.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T23:35:15.517 INFO:teuthology.orchestra.run.smithi031.stdout:The following additional packages will be installed: 2024-09-21T23:35:15.517 INFO:teuthology.orchestra.run.smithi031.stdout: linux-generic linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-21T23:35:15.517 INFO:teuthology.orchestra.run.smithi031.stdout: linux-headers-generic linux-image-5.15.0-122-generic 2024-09-21T23:35:15.517 INFO:teuthology.orchestra.run.smithi031.stdout: linux-modules-5.15.0-122-generic linux-modules-extra-5.15.0-122-generic 2024-09-21T23:35:15.518 INFO:teuthology.orchestra.run.smithi031.stdout:Suggested packages: 2024-09-21T23:35:15.519 INFO:teuthology.orchestra.run.smithi031.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-21T23:35:15.519 INFO:teuthology.orchestra.run.smithi031.stdout:Recommended packages: 2024-09-21T23:35:15.519 INFO:teuthology.orchestra.run.smithi031.stdout: thermald 2024-09-21T23:35:15.564 INFO:teuthology.orchestra.run.smithi031.stdout:The following NEW packages will be installed: 2024-09-21T23:35:15.564 INFO:teuthology.orchestra.run.smithi031.stdout: linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-21T23:35:15.564 INFO:teuthology.orchestra.run.smithi031.stdout: linux-image-5.15.0-122-generic linux-modules-5.15.0-122-generic 2024-09-21T23:35:15.565 INFO:teuthology.orchestra.run.smithi031.stdout: linux-modules-extra-5.15.0-122-generic 2024-09-21T23:35:15.566 INFO:teuthology.orchestra.run.smithi031.stdout:The following packages will be upgraded: 2024-09-21T23:35:15.566 INFO:teuthology.orchestra.run.smithi031.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-21T23:35:15.573 INFO:teuthology.orchestra.run.smithi060.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T23:35:15.573 INFO:teuthology.orchestra.run.smithi060.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T23:35:15.573 INFO:teuthology.orchestra.run.smithi060.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T23:35:15.574 INFO:teuthology.orchestra.run.smithi060.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T23:35:15.574 INFO:teuthology.orchestra.run.smithi060.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-21T23:35:15.574 INFO:teuthology.orchestra.run.smithi060.stdout: python2.7-minimal 2024-09-21T23:35:15.574 INFO:teuthology.orchestra.run.smithi060.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T23:35:15.576 INFO:teuthology.orchestra.run.smithi060.stdout:The following additional packages will be installed: 2024-09-21T23:35:15.576 INFO:teuthology.orchestra.run.smithi060.stdout: linux-generic linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-21T23:35:15.576 INFO:teuthology.orchestra.run.smithi060.stdout: linux-headers-generic linux-image-5.15.0-122-generic 2024-09-21T23:35:15.576 INFO:teuthology.orchestra.run.smithi060.stdout: linux-modules-5.15.0-122-generic linux-modules-extra-5.15.0-122-generic 2024-09-21T23:35:15.578 INFO:teuthology.orchestra.run.smithi060.stdout:Suggested packages: 2024-09-21T23:35:15.578 INFO:teuthology.orchestra.run.smithi060.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-21T23:35:15.578 INFO:teuthology.orchestra.run.smithi060.stdout:Recommended packages: 2024-09-21T23:35:15.578 INFO:teuthology.orchestra.run.smithi060.stdout: thermald 2024-09-21T23:35:15.639 INFO:teuthology.orchestra.run.smithi060.stdout:The following NEW packages will be installed: 2024-09-21T23:35:15.639 INFO:teuthology.orchestra.run.smithi060.stdout: linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-21T23:35:15.639 INFO:teuthology.orchestra.run.smithi060.stdout: linux-image-5.15.0-122-generic linux-modules-5.15.0-122-generic 2024-09-21T23:35:15.639 INFO:teuthology.orchestra.run.smithi060.stdout: linux-modules-extra-5.15.0-122-generic 2024-09-21T23:35:15.640 INFO:teuthology.orchestra.run.smithi060.stdout:The following packages will be upgraded: 2024-09-21T23:35:15.641 INFO:teuthology.orchestra.run.smithi060.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-21T23:35:15.746 INFO:teuthology.orchestra.run.smithi031.stdout:3 upgraded, 5 newly installed, 0 to remove and 345 not upgraded. 2024-09-21T23:35:15.747 INFO:teuthology.orchestra.run.smithi031.stdout:Need to get 113 MB of archives. 2024-09-21T23:35:15.747 INFO:teuthology.orchestra.run.smithi031.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-21T23:35:15.747 INFO:teuthology.orchestra.run.smithi031.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-122-generic amd64 5.15.0-122.132 [22.7 MB] 2024-09-21T23:35:15.823 INFO:teuthology.orchestra.run.smithi060.stdout:3 upgraded, 5 newly installed, 0 to remove and 345 not upgraded. 2024-09-21T23:35:15.823 INFO:teuthology.orchestra.run.smithi060.stdout:Need to get 113 MB of archives. 2024-09-21T23:35:15.823 INFO:teuthology.orchestra.run.smithi060.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-21T23:35:15.823 INFO:teuthology.orchestra.run.smithi060.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-122-generic amd64 5.15.0-122.132 [22.7 MB] 2024-09-21T23:35:17.196 INFO:teuthology.orchestra.run.smithi031.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-122-generic amd64 5.15.0-122.132 [11.5 MB] 2024-09-21T23:35:17.224 INFO:teuthology.orchestra.run.smithi060.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-122-generic amd64 5.15.0-122.132 [11.5 MB] 2024-09-21T23:35:17.494 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-122-generic amd64 5.15.0-122.132 [63.9 MB] 2024-09-21T23:35:17.553 INFO:teuthology.orchestra.run.smithi031.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-122-generic amd64 5.15.0-122.132 [63.9 MB] 2024-09-21T23:35:18.379 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.122.122 [1,692 B] 2024-09-21T23:35:18.397 INFO:teuthology.orchestra.run.smithi060.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.122.122 [2,436 B] 2024-09-21T23:35:18.415 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122 all 5.15.0-122.132 [12.3 MB] 2024-09-21T23:35:18.659 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122-generic amd64 5.15.0-122.132 [2,827 kB] 2024-09-21T23:35:18.708 INFO:teuthology.orchestra.run.smithi060.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.122.122 [2,294 B] 2024-09-21T23:35:19.167 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 113 MB in 3s (37.0 MB/s) 2024-09-21T23:35:19.343 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-modules-5.15.0-122-generic. 2024-09-21T23:35:19.812 INFO:teuthology.orchestra.run.smithi031.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.122.122 [1,692 B] 2024-09-21T23:35:19.812 INFO:teuthology.orchestra.run.smithi031.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.122.122 [2,436 B] 2024-09-21T23:35:19.812 INFO:teuthology.orchestra.run.smithi031.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122 all 5.15.0-122.132 [12.3 MB] 2024-09-21T23:35:20.169 INFO:teuthology.orchestra.run.smithi031.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122-generic amd64 5.15.0-122.132 [2,827 kB] 2024-09-21T23:35:20.253 INFO:teuthology.orchestra.run.smithi031.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.122.122 [2,294 B] 2024-09-21T23:35:20.684 INFO:teuthology.orchestra.run.smithi031.stdout:Fetched 113 MB in 5s (24.2 MB/s) 2024-09-21T23:35:20.843 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package linux-modules-5.15.0-122-generic. 2024-09-21T23:35:21.389 INFO:teuthology.orchestra.run.smithi060.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-21T23:35:21.393 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../0-linux-modules-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T23:35:21.482 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T23:35:22.829 INFO:teuthology.orchestra.run.smithi031.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-21T23:35:22.833 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../0-linux-modules-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T23:35:23.031 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T23:35:24.034 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-image-5.15.0-122-generic. 2024-09-21T23:35:24.055 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../1-linux-image-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T23:35:24.129 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T23:35:24.545 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-122-generic. 2024-09-21T23:35:24.566 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T23:35:24.595 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T23:35:25.081 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package linux-image-5.15.0-122-generic. 2024-09-21T23:35:25.103 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../1-linux-image-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T23:35:25.182 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T23:35:25.693 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-122-generic. 2024-09-21T23:35:25.713 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T23:35:25.750 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T23:35:30.941 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../3-linux-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T23:35:31.091 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T23:35:31.441 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T23:35:31.553 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-image-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T23:35:31.822 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-headers-5.15.0-122. 2024-09-21T23:35:31.845 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../5-linux-headers-5.15.0-122_5.15.0-122.132_all.deb ... 2024-09-21T23:35:31.889 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-21T23:35:32.132 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../3-linux-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T23:35:32.258 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking linux-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T23:35:32.600 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T23:35:32.728 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking linux-image-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T23:35:33.023 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package linux-headers-5.15.0-122. 2024-09-21T23:35:33.034 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../5-linux-headers-5.15.0-122_5.15.0-122.132_all.deb ... 2024-09-21T23:35:33.064 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-21T23:35:37.845 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-headers-5.15.0-122-generic. 2024-09-21T23:35:37.882 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../6-linux-headers-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T23:35:37.919 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T23:35:38.339 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package linux-headers-5.15.0-122-generic. 2024-09-21T23:35:38.375 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../6-linux-headers-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T23:35:38.413 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T23:35:40.747 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T23:35:40.856 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-headers-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T23:35:41.189 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-21T23:35:41.275 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T23:35:41.310 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T23:35:41.400 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking linux-headers-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T23:35:41.436 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-headers-generic (5.15.0.122.122) ... 2024-09-21T23:35:41.565 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T23:35:42.660 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-21T23:35:42.789 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T23:35:42.880 INFO:teuthology.orchestra.run.smithi060.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-21T23:35:42.880 INFO:teuthology.orchestra.run.smithi060.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-21T23:35:42.880 INFO:teuthology.orchestra.run.smithi060.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-122-generic 2024-09-21T23:35:42.880 INFO:teuthology.orchestra.run.smithi060.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-122-generic 2024-09-21T23:35:42.915 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up linux-headers-generic (5.15.0.122.122) ... 2024-09-21T23:35:43.036 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T23:35:43.068 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T23:35:44.389 INFO:teuthology.orchestra.run.smithi031.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-21T23:35:44.389 INFO:teuthology.orchestra.run.smithi031.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-21T23:35:44.389 INFO:teuthology.orchestra.run.smithi031.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-122-generic 2024-09-21T23:35:44.389 INFO:teuthology.orchestra.run.smithi031.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-122-generic 2024-09-21T23:35:44.546 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T23:35:44.561 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-image-generic (5.15.0.122.122) ... 2024-09-21T23:35:44.687 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-generic (5.15.0.122.122) ... 2024-09-21T23:35:44.813 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T23:35:46.072 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up linux-image-generic (5.15.0.122.122) ... 2024-09-21T23:35:46.198 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up linux-generic (5.15.0.122.122) ... 2024-09-21T23:35:46.232 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T23:35:46.282 INFO:teuthology.orchestra.run.smithi060.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-21T23:35:46.282 INFO:teuthology.orchestra.run.smithi060.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-122-generic 2024-09-21T23:35:46.324 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T23:35:48.543 INFO:teuthology.orchestra.run.smithi031.stdout:Processing triggers for linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T23:35:48.636 INFO:teuthology.orchestra.run.smithi031.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-21T23:35:48.636 INFO:teuthology.orchestra.run.smithi031.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-122-generic 2024-09-21T23:36:00.816 INFO:teuthology.orchestra.run.smithi060.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-21T23:36:00.816 INFO:teuthology.orchestra.run.smithi060.stdout:Sourcing file `/etc/default/grub' 2024-09-21T23:36:00.826 INFO:teuthology.orchestra.run.smithi060.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T23:36:00.846 INFO:teuthology.orchestra.run.smithi060.stdout:Generating grub configuration file ... 2024-09-21T23:36:01.137 INFO:teuthology.orchestra.run.smithi060.stdout:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-21T23:36:01.161 INFO:teuthology.orchestra.run.smithi060.stdout:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-21T23:36:01.451 INFO:teuthology.orchestra.run.smithi060.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-21T23:36:01.456 INFO:teuthology.orchestra.run.smithi060.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-21T23:36:01.535 INFO:teuthology.orchestra.run.smithi060.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-21T23:36:01.541 INFO:teuthology.orchestra.run.smithi060.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-21T23:36:01.719 INFO:teuthology.orchestra.run.smithi060.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-21T23:36:01.719 INFO:teuthology.orchestra.run.smithi060.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-21T23:36:01.719 INFO:teuthology.orchestra.run.smithi060.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-21T23:36:01.760 INFO:teuthology.orchestra.run.smithi060.stdout:done 2024-09-21T23:36:02.226 DEBUG:teuthology.orchestra.run.smithi060:> dpkg -s linux-image-generic 2024-09-21T23:36:02.250 INFO:teuthology.orchestra.run.smithi060.stdout:Package: linux-image-generic 2024-09-21T23:36:02.250 INFO:teuthology.orchestra.run.smithi060.stdout:Status: install ok installed 2024-09-21T23:36:02.250 INFO:teuthology.orchestra.run.smithi060.stdout:Priority: optional 2024-09-21T23:36:02.250 INFO:teuthology.orchestra.run.smithi060.stdout:Section: kernel 2024-09-21T23:36:02.250 INFO:teuthology.orchestra.run.smithi060.stdout:Installed-Size: 21 2024-09-21T23:36:02.250 INFO:teuthology.orchestra.run.smithi060.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T23:36:02.250 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: amd64 2024-09-21T23:36:02.251 INFO:teuthology.orchestra.run.smithi060.stdout:Source: linux-meta 2024-09-21T23:36:02.251 INFO:teuthology.orchestra.run.smithi060.stdout:Version: 5.15.0.122.122 2024-09-21T23:36:02.251 INFO:teuthology.orchestra.run.smithi060.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-21T23:36:02.251 INFO:teuthology.orchestra.run.smithi060.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T23:36:02.251 INFO:teuthology.orchestra.run.smithi060.stdout:Recommends: thermald 2024-09-21T23:36:02.251 INFO:teuthology.orchestra.run.smithi060.stdout:Description: Generic Linux kernel image 2024-09-21T23:36:02.251 INFO:teuthology.orchestra.run.smithi060.stdout: This package will always depend on the latest generic kernel image 2024-09-21T23:36:02.251 INFO:teuthology.orchestra.run.smithi060.stdout: available. 2024-09-21T23:36:02.251 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-122-generic 2024-09-21T23:36:02.251 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T23:36:02.251 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T23:36:02.251 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-21T23:36:02.251 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi060.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T23:36:02.252 DEBUG:teuthology.orchestra.run.smithi060:> sudo apt-get clean 2024-09-21T23:36:02.410 DEBUG:teuthology.orchestra.run.smithi060:> sudo apt-get update 2024-09-21T23:36:02.660 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T23:36:02.660 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-21T23:36:02.732 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-21T23:36:02.814 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-21T23:36:02.986 INFO:teuthology.orchestra.run.smithi031.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-21T23:36:02.986 INFO:teuthology.orchestra.run.smithi031.stdout:Sourcing file `/etc/default/grub' 2024-09-21T23:36:02.999 INFO:teuthology.orchestra.run.smithi031.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T23:36:03.018 INFO:teuthology.orchestra.run.smithi031.stdout:Generating grub configuration file ... 2024-09-21T23:36:03.281 INFO:teuthology.orchestra.run.smithi031.stdout:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-21T23:36:03.297 INFO:teuthology.orchestra.run.smithi031.stdout:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-21T23:36:03.603 INFO:teuthology.orchestra.run.smithi031.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-21T23:36:03.609 INFO:teuthology.orchestra.run.smithi031.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-21T23:36:03.680 INFO:teuthology.orchestra.run.smithi031.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-21T23:36:03.686 INFO:teuthology.orchestra.run.smithi031.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-21T23:36:03.835 INFO:teuthology.orchestra.run.smithi031.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-21T23:36:03.835 INFO:teuthology.orchestra.run.smithi031.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-21T23:36:03.835 INFO:teuthology.orchestra.run.smithi031.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-21T23:36:03.879 INFO:teuthology.orchestra.run.smithi031.stdout:done 2024-09-21T23:36:04.114 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2024-09-21T23:36:04.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T23:36:04.199 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2024-09-21T23:36:04.338 DEBUG:teuthology.orchestra.run.smithi031:> dpkg -s linux-image-generic 2024-09-21T23:36:04.362 INFO:teuthology.orchestra.run.smithi031.stdout:Package: linux-image-generic 2024-09-21T23:36:04.362 INFO:teuthology.orchestra.run.smithi031.stdout:Status: install ok installed 2024-09-21T23:36:04.362 INFO:teuthology.orchestra.run.smithi031.stdout:Priority: optional 2024-09-21T23:36:04.363 INFO:teuthology.orchestra.run.smithi031.stdout:Section: kernel 2024-09-21T23:36:04.363 INFO:teuthology.orchestra.run.smithi031.stdout:Installed-Size: 21 2024-09-21T23:36:04.363 INFO:teuthology.orchestra.run.smithi031.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T23:36:04.363 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: amd64 2024-09-21T23:36:04.363 INFO:teuthology.orchestra.run.smithi031.stdout:Source: linux-meta 2024-09-21T23:36:04.363 INFO:teuthology.orchestra.run.smithi031.stdout:Version: 5.15.0.122.122 2024-09-21T23:36:04.363 INFO:teuthology.orchestra.run.smithi031.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-21T23:36:04.363 INFO:teuthology.orchestra.run.smithi031.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T23:36:04.363 INFO:teuthology.orchestra.run.smithi031.stdout:Recommends: thermald 2024-09-21T23:36:04.363 INFO:teuthology.orchestra.run.smithi031.stdout:Description: Generic Linux kernel image 2024-09-21T23:36:04.363 INFO:teuthology.orchestra.run.smithi031.stdout: This package will always depend on the latest generic kernel image 2024-09-21T23:36:04.363 INFO:teuthology.orchestra.run.smithi031.stdout: available. 2024-09-21T23:36:04.364 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-122-generic 2024-09-21T23:36:04.364 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T23:36:04.364 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T23:36:04.364 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-21T23:36:04.364 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi031.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T23:36:04.364 DEBUG:teuthology.orchestra.run.smithi031:> sudo apt-get clean 2024-09-21T23:36:04.415 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2024-09-21T23:36:04.416 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2024-09-21T23:36:04.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo apt-get update 2024-09-21T23:36:04.612 INFO:teuthology.orchestra.run.smithi060.stdout:linux-image-generic is already the newest version (5.15.0.122.122). 2024-09-21T23:36:04.613 INFO:teuthology.orchestra.run.smithi060.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T23:36:04.613 INFO:teuthology.orchestra.run.smithi060.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T23:36:04.613 INFO:teuthology.orchestra.run.smithi060.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T23:36:04.613 INFO:teuthology.orchestra.run.smithi060.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T23:36:04.613 INFO:teuthology.orchestra.run.smithi060.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T23:36:04.613 INFO:teuthology.orchestra.run.smithi060.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T23:36:04.613 INFO:teuthology.orchestra.run.smithi060.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T23:36:04.657 INFO:teuthology.orchestra.run.smithi060.stdout:0 upgraded, 0 newly installed, 0 to remove and 345 not upgraded. 2024-09-21T23:36:04.659 DEBUG:teuthology.orchestra.run.smithi060:> dpkg -s linux-image-generic 2024-09-21T23:36:04.675 INFO:teuthology.orchestra.run.smithi060.stdout:Package: linux-image-generic 2024-09-21T23:36:04.675 INFO:teuthology.orchestra.run.smithi060.stdout:Status: install ok installed 2024-09-21T23:36:04.676 INFO:teuthology.orchestra.run.smithi060.stdout:Priority: optional 2024-09-21T23:36:04.676 INFO:teuthology.orchestra.run.smithi060.stdout:Section: kernel 2024-09-21T23:36:04.676 INFO:teuthology.orchestra.run.smithi060.stdout:Installed-Size: 21 2024-09-21T23:36:04.676 INFO:teuthology.orchestra.run.smithi060.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T23:36:04.676 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: amd64 2024-09-21T23:36:04.676 INFO:teuthology.orchestra.run.smithi060.stdout:Source: linux-meta 2024-09-21T23:36:04.676 INFO:teuthology.orchestra.run.smithi060.stdout:Version: 5.15.0.122.122 2024-09-21T23:36:04.676 INFO:teuthology.orchestra.run.smithi060.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-21T23:36:04.676 INFO:teuthology.orchestra.run.smithi060.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T23:36:04.676 INFO:teuthology.orchestra.run.smithi060.stdout:Recommends: thermald 2024-09-21T23:36:04.676 INFO:teuthology.orchestra.run.smithi060.stdout:Description: Generic Linux kernel image 2024-09-21T23:36:04.676 INFO:teuthology.orchestra.run.smithi060.stdout: This package will always depend on the latest generic kernel image 2024-09-21T23:36:04.676 INFO:teuthology.orchestra.run.smithi060.stdout: available. 2024-09-21T23:36:04.677 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-09-21T23:36:04.677 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T23:36:04.685 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-21T23:36:04.702 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-21T23:36:04.723 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.JHf51SRfK5 2024-09-21T23:36:04.724 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /boot/grub/grub.cfg /tmp/tmp.JHf51SRfK5 2024-09-21T23:36:04.796 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.JHf51SRfK5 2024-09-21T23:36:04.798 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-21T23:36:04.914 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.JHf51SRfK5 is 10KB 2024-09-21T23:36:04.929 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.JHf51SRfK5 2024-09-21T23:36:04.935 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-21T23:36:04.987 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:36:04.987 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-21T23:36:04.987 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-21T23:36:05.055 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-122-generic 2024-09-21T23:36:05.055 DEBUG:teuthology.orchestra.run.smithi060:> sudo update-grub 2024-09-21T23:36:06.104 INFO:teuthology.orchestra.run.smithi031.stdout:Reading package lists... 2024-09-21T23:36:06.123 DEBUG:teuthology.orchestra.run.smithi031:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T23:36:06.178 INFO:teuthology.orchestra.run.smithi031.stdout:Reading package lists... 2024-09-21T23:36:06.271 INFO:teuthology.orchestra.run.smithi060.stderr:Sourcing file `/etc/default/grub' 2024-09-21T23:36:06.272 INFO:teuthology.orchestra.run.smithi060.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T23:36:06.275 INFO:teuthology.orchestra.run.smithi060.stderr:Generating grub configuration file ... 2024-09-21T23:36:06.369 INFO:teuthology.orchestra.run.smithi031.stdout:Building dependency tree... 2024-09-21T23:36:06.370 INFO:teuthology.orchestra.run.smithi031.stdout:Reading state information... 2024-09-21T23:36:06.438 INFO:teuthology.orchestra.run.smithi060.stderr:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-21T23:36:06.445 INFO:teuthology.orchestra.run.smithi060.stderr:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-21T23:36:06.546 INFO:teuthology.orchestra.run.smithi031.stdout:linux-image-generic is already the newest version (5.15.0.122.122). 2024-09-21T23:36:06.546 INFO:teuthology.orchestra.run.smithi031.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T23:36:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T23:36:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T23:36:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T23:36:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T23:36:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T23:36:06.547 INFO:teuthology.orchestra.run.smithi031.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T23:36:06.585 INFO:teuthology.orchestra.run.smithi031.stdout:0 upgraded, 0 newly installed, 0 to remove and 345 not upgraded. 2024-09-21T23:36:06.587 DEBUG:teuthology.orchestra.run.smithi031:> dpkg -s linux-image-generic 2024-09-21T23:36:06.602 INFO:teuthology.orchestra.run.smithi031.stdout:Package: linux-image-generic 2024-09-21T23:36:06.603 INFO:teuthology.orchestra.run.smithi031.stdout:Status: install ok installed 2024-09-21T23:36:06.603 INFO:teuthology.orchestra.run.smithi031.stdout:Priority: optional 2024-09-21T23:36:06.603 INFO:teuthology.orchestra.run.smithi031.stdout:Section: kernel 2024-09-21T23:36:06.603 INFO:teuthology.orchestra.run.smithi031.stdout:Installed-Size: 21 2024-09-21T23:36:06.603 INFO:teuthology.orchestra.run.smithi031.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T23:36:06.603 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: amd64 2024-09-21T23:36:06.603 INFO:teuthology.orchestra.run.smithi031.stdout:Source: linux-meta 2024-09-21T23:36:06.603 INFO:teuthology.orchestra.run.smithi031.stdout:Version: 5.15.0.122.122 2024-09-21T23:36:06.603 INFO:teuthology.orchestra.run.smithi031.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-21T23:36:06.603 INFO:teuthology.orchestra.run.smithi031.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T23:36:06.603 INFO:teuthology.orchestra.run.smithi031.stdout:Recommends: thermald 2024-09-21T23:36:06.603 INFO:teuthology.orchestra.run.smithi031.stdout:Description: Generic Linux kernel image 2024-09-21T23:36:06.604 INFO:teuthology.orchestra.run.smithi031.stdout: This package will always depend on the latest generic kernel image 2024-09-21T23:36:06.604 INFO:teuthology.orchestra.run.smithi031.stdout: available. 2024-09-21T23:36:06.604 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-09-21T23:36:06.650 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.8xWWogVPc2 2024-09-21T23:36:06.650 DEBUG:teuthology.orchestra.run.smithi031:> sudo cp /boot/grub/grub.cfg /tmp/tmp.8xWWogVPc2 2024-09-21T23:36:06.674 INFO:teuthology.orchestra.run.smithi060.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-21T23:36:06.680 INFO:teuthology.orchestra.run.smithi060.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-21T23:36:06.703 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.8xWWogVPc2 2024-09-21T23:36:06.727 INFO:teuthology.orchestra.run.smithi060.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-21T23:36:06.733 INFO:teuthology.orchestra.run.smithi060.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-21T23:36:06.816 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.8xWWogVPc2 is 10KB 2024-09-21T23:36:06.826 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.8xWWogVPc2 2024-09-21T23:36:06.831 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-21T23:36:06.878 INFO:teuthology.orchestra.run.smithi060.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-21T23:36:06.878 INFO:teuthology.orchestra.run.smithi060.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-21T23:36:06.878 INFO:teuthology.orchestra.run.smithi060.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-21T23:36:06.885 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:36:06.885 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-21T23:36:06.885 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-21T23:36:06.897 INFO:teuthology.orchestra.run.smithi060.stderr:done 2024-09-21T23:36:06.906 DEBUG:teuthology.orchestra.run.smithi060:> sudo shutdown -r now 2024-09-21T23:36:06.946 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-122-generic 2024-09-21T23:36:06.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo update-grub 2024-09-21T23:36:08.237 INFO:teuthology.orchestra.run.smithi031.stderr:Sourcing file `/etc/default/grub' 2024-09-21T23:36:08.238 INFO:teuthology.orchestra.run.smithi031.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T23:36:08.241 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2024-09-21T23:36:08.422 INFO:teuthology.orchestra.run.smithi031.stderr:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-21T23:36:08.428 INFO:teuthology.orchestra.run.smithi031.stderr:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-21T23:36:08.648 INFO:teuthology.orchestra.run.smithi031.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-21T23:36:08.653 INFO:teuthology.orchestra.run.smithi031.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-21T23:36:08.694 INFO:teuthology.orchestra.run.smithi031.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-21T23:36:08.700 INFO:teuthology.orchestra.run.smithi031.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-21T23:36:08.822 INFO:teuthology.orchestra.run.smithi031.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-21T23:36:08.822 INFO:teuthology.orchestra.run.smithi031.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-21T23:36:08.823 INFO:teuthology.orchestra.run.smithi031.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-21T23:36:08.842 INFO:teuthology.orchestra.run.smithi031.stderr:done 2024-09-21T23:36:08.844 DEBUG:teuthology.orchestra.run.smithi031:> sudo shutdown -r now 2024-09-21T23:36:36.932 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-21T23:36:36.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-21T23:36:36.933 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T23:36:38.847 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-21T23:36:38.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-21T23:36:38.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T23:36:55.279 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.60 2024-09-21T23:36:57.327 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-09-21T23:37:04.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-21T23:37:04.287 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T23:37:06.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-21T23:37:06.330 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T23:38:04.347 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T23:38:06.334 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T23:38:16.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-21T23:38:16.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T23:38:16.561 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-09-21T23:38:17.162 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-21T23:38:17.162 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.15.0-122-generic"... 2024-09-21T23:38:17.163 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-09-21T23:38:17.168 INFO:teuthology.orchestra.run.smithi060.stdout:5.15.0-122-generic 2024-09-21T23:38:17.168 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-122-generic vs 5.15.0-122-generic 2024-09-21T23:38:17.168 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T23:38:17.168 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-21T23:38:18.170 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-21T23:38:18.170 DEBUG:teuthology.orchestra.run.smithi060:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T23:38:18.239 INFO:teuthology.orchestra.run.smithi060.stdout:ttyS1 2024-09-21T23:38:18.276 DEBUG:teuthology.parallel:result is None 2024-09-21T23:38:18.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-21T23:38:18.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T23:38:18.564 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-09-21T23:38:19.235 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-21T23:38:19.235 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.15.0-122-generic"... 2024-09-21T23:38:19.235 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-09-21T23:38:19.280 INFO:teuthology.orchestra.run.smithi031.stdout:5.15.0-122-generic 2024-09-21T23:38:19.281 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-122-generic vs 5.15.0-122-generic 2024-09-21T23:38:19.281 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T23:38:19.281 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-21T23:38:20.281 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-21T23:38:20.282 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T23:38:20.358 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2024-09-21T23:38:20.386 DEBUG:teuthology.parallel:result is None 2024-09-21T23:38:20.386 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T23:38:20.393 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T23:38:20.393 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T23:38:20.396 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T23:38:20.402 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T23:38:20.593 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T23:38:20.665 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T23:38:20.665 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T23:38:20.668 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T23:38:20.685 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T23:38:20.770 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T23:38:20.770 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T23:38:20.773 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T23:38:20.794 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T23:38:20.795 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T23:38:20.802 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T23:38:20.808 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T23:38:20.809 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T23:38:20.894 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T23:38:20.894 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T23:38:20.897 DEBUG:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T23:38:20.911 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T23:38:20.968 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T23:38:20.969 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T23:38:20.971 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T23:38:20.977 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T23:38:21.021 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T23:38:21.069 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:38:21.069 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T23:38:21.133 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T23:38:21.139 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T23:38:21.187 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:38:21.187 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T23:38:21.251 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2024-09-21T23:38:21.254 DEBUG:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2024-09-21T23:38:21.327 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T23:38:21.335 INFO:teuthology.task.internal:Starting timer... 2024-09-21T23:38:21.335 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T23:38:21.418 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T23:38:21.611 DEBUG:teuthology.task.selinux:Excluding smithi031: OS 'ubuntu' does not support SELinux 2024-09-21T23:38:21.612 DEBUG:teuthology.task.selinux:Excluding smithi060: OS 'ubuntu' does not support SELinux 2024-09-21T23:38:21.612 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T23:38:21.612 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-21T23:38:21.612 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T23:38:21.612 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T23:38:21.752 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T23:38:21.764 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T23:38:21.764 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com,smithi060.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T23:45:58.738 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com'), Remote(name='ubuntu@smithi060.front.sepia.ceph.com')] 2024-09-21T23:45:58.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-21T23:45:58.740 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T23:45:58.822 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-09-21T23:45:58.898 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-21T23:45:58.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-21T23:45:58.899 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T23:45:58.979 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-09-21T23:45:59.050 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-21T23:45:59.050 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T23:45:59.060 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T23:45:59.060 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T23:45:59.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T23:45:59.063 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T23:45:59.063 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T23:45:59.093 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-21T23:45:59.093 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: Command line: ntpd -gq 2024-09-21T23:45:59.093 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: ---------------------------------------------------- 2024-09-21T23:45:59.093 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: ntp-4 is maintained by Network Time Foundation, 2024-09-21T23:45:59.093 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-21T23:45:59.093 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: corporation. Support and training for ntp-4 are 2024-09-21T23:45:59.093 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: available at https://www.nwtime.org/support 2024-09-21T23:45:59.093 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: ---------------------------------------------------- 2024-09-21T23:45:59.093 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: proto: precision = 0.046 usec (-24) 2024-09-21T23:45:59.094 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: basedate set to 2022-02-04 2024-09-21T23:45:59.094 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: gps base set to 2022-02-06 (week 2196) 2024-09-21T23:45:59.094 INFO:teuthology.orchestra.run.smithi031.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T23:45:59.094 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T23:45:59.095 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: restrict ::: KOD does nothing without LIMITED. 2024-09-21T23:45:59.095 INFO:teuthology.orchestra.run.smithi031.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-21T23:45:59.096 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: Listen and drop on 0 v6wildcard [::]:123 2024-09-21T23:45:59.096 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-21T23:45:59.096 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: Listen normally on 2 lo 127.0.0.1:123 2024-09-21T23:45:59.096 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: Listen normally on 3 ens1f0 172.21.15.31:123 2024-09-21T23:45:59.096 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: Listen normally on 4 lo [::1]:123 2024-09-21T23:45:59.096 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fea5:caba%4]:123 2024-09-21T23:45:59.096 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:45:59 ntpd[18357]: Listening on routing socket on fd #22 for interface updates 2024-09-21T23:45:59.120 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-21T23:45:59.120 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: Command line: ntpd -gq 2024-09-21T23:45:59.120 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: ---------------------------------------------------- 2024-09-21T23:45:59.120 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: ntp-4 is maintained by Network Time Foundation, 2024-09-21T23:45:59.120 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-21T23:45:59.120 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: corporation. Support and training for ntp-4 are 2024-09-21T23:45:59.120 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: available at https://www.nwtime.org/support 2024-09-21T23:45:59.121 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: ---------------------------------------------------- 2024-09-21T23:45:59.121 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: proto: precision = 0.063 usec (-24) 2024-09-21T23:45:59.121 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: basedate set to 2022-02-04 2024-09-21T23:45:59.121 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: gps base set to 2022-02-06 (week 2196) 2024-09-21T23:45:59.121 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T23:45:59.122 INFO:teuthology.orchestra.run.smithi060.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T23:45:59.122 INFO:teuthology.orchestra.run.smithi060.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-21T23:45:59.122 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: restrict ::: KOD does nothing without LIMITED. 2024-09-21T23:45:59.123 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: Listen and drop on 0 v6wildcard [::]:123 2024-09-21T23:45:59.123 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-21T23:45:59.123 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: Listen normally on 2 lo 127.0.0.1:123 2024-09-21T23:45:59.123 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: Listen normally on 3 ens1f0 172.21.15.60:123 2024-09-21T23:45:59.123 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: Listen normally on 4 lo [::1]:123 2024-09-21T23:45:59.123 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1560%4]:123 2024-09-21T23:45:59.123 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:45:59 ntpd[18365]: Listening on routing socket on fd #22 for interface updates 2024-09-21T23:46:00.093 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:00 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:00.093 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:00 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:00.093 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:00 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:00.094 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:00 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:00.094 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:00 ntpd[18357]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T23:46:00.094 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:00 ntpd[18357]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T23:46:00.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:00 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:00.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:00 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:00.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:00 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:00.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:00 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:00.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:00 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:00.123 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:00 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:00.123 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:00 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T23:46:00.123 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:00 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T23:46:00.123 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:00 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:00.123 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:00 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:00.123 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:00 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T23:46:00.123 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:00 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T23:46:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:01 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:01 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:02.094 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:02 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:02.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:02 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:02.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:02 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:02.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:02 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:02.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:02 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:02.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:02 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:02.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:02 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:02.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:02 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:02.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:02 ntpd[18357]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T23:46:02.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:02 ntpd[18357]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T23:46:02.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:02 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:02.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:02 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:02.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:02 ntpd[18357]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T23:46:02.096 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:02 ntpd[18357]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T23:46:02.121 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:02 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:02.121 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:02 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:02.121 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:02 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:02.121 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:02 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:02.121 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:02 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:02.121 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:02 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:02.121 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:02 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T23:46:02.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:02 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T23:46:02.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:02 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:02.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:02 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:02.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:02 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T23:46:02.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:02 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T23:46:04.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:04 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:04.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:04 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:04.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:04 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:04.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:04 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:04.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:04 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:04.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:04 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:04.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:04 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:04.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:04 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:04.096 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:04 ntpd[18357]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T23:46:04.096 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:04 ntpd[18357]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T23:46:04.096 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:04 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:04.096 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:04 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:04.096 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:04 ntpd[18357]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T23:46:04.096 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:04 ntpd[18357]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T23:46:04.096 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:04 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:04.096 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:04 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:04.096 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:04 ntpd[18357]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T23:46:04.096 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:04 ntpd[18357]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T23:46:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:04 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:04 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:04 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:04 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:04 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:04 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:04 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:04 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:04 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T23:46:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:04 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T23:46:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:04 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:04 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:04 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T23:46:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:04 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T23:46:04.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:04 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:04.123 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:04 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:04.123 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:04 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T23:46:04.123 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:04 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T23:46:06.094 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:06 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:06.094 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:06 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:06.094 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:06 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:06.094 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:06 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:06.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:06 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:06.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:06 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:06.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:06 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:06.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:06 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:06.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:06 ntpd[18357]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T23:46:06.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:06 ntpd[18357]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T23:46:06.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:06 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:06.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:06 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:06.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:06 ntpd[18357]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T23:46:06.095 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:06 ntpd[18357]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T23:46:06.095 INFO:teuthology.orchestra.run.smithi031.stdout:21 Sep 23:46:06 ntpd[18357]: ntpd: time slew +0.002039 s 2024-09-21T23:46:06.095 INFO:teuthology.orchestra.run.smithi031.stdout:ntpd: time slew +0.002039s 2024-09-21T23:46:06.096 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:06 ntpd[18357]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-21T23:46:06.096 INFO:teuthology.orchestra.run.smithi031.stderr:21 Sep 23:46:06 ntpd[18357]: can't open /var/log/ntpstats/loopstats.20240921: Permission denied 2024-09-21T23:46:06.121 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:06 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:06.121 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:06 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:06 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:06 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:06 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:06 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:06 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:06 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:06 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T23:46:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:06 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T23:46:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:06 ntpd[18365]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T23:46:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:06 ntpd[18365]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T23:46:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:06 ntpd[18365]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T23:46:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:06 ntpd[18365]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T23:46:06.123 INFO:teuthology.orchestra.run.smithi060.stdout:21 Sep 23:46:06 ntpd[18365]: ntpd: time slew +0.002849 s 2024-09-21T23:46:06.123 INFO:teuthology.orchestra.run.smithi060.stdout:ntpd: time slew +0.002849s 2024-09-21T23:46:06.123 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:06 ntpd[18365]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-21T23:46:06.123 INFO:teuthology.orchestra.run.smithi060.stderr:21 Sep 23:46:06 ntpd[18365]: can't open /var/log/ntpstats/loopstats.20240921: Permission denied 2024-09-21T23:46:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T23:46:06.146 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================== 2024-09-21T23:46:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T23:46:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T23:46:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T23:46:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T23:46:06.171 INFO:teuthology.orchestra.run.smithi060.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T23:46:06.172 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================== 2024-09-21T23:46:06.172 INFO:teuthology.orchestra.run.smithi060.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T23:46:06.172 INFO:teuthology.orchestra.run.smithi060.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T23:46:06.172 INFO:teuthology.orchestra.run.smithi060.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T23:46:06.172 INFO:teuthology.orchestra.run.smithi060.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T23:46:06.172 INFO:teuthology.run_tasks:Running task install... 2024-09-21T23:46:06.180 DEBUG:teuthology.task.install:project ceph 2024-09-21T23:46:06.180 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}, 'extra_system_packages': ['pv']} 2024-09-21T23:46:06.180 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'extra_system_packages': ['pv']} 2024-09-21T23:46:06.181 INFO:teuthology.task.install:Using flavor: default 2024-09-21T23:46:06.186 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T23:46:06.186 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-09-21T23:46:06.187 DEBUG:teuthology.orchestra.run.smithi031:> sudo apt-key list | grep Ceph 2024-09-21T23:46:06.187 DEBUG:teuthology.orchestra.run.smithi060:> sudo apt-key list | grep Ceph 2024-09-21T23:46:06.252 INFO:teuthology.orchestra.run.smithi031.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-21T23:46:06.271 INFO:teuthology.orchestra.run.smithi060.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-21T23:46:06.281 INFO:teuthology.orchestra.run.smithi031.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-21T23:46:06.282 INFO:teuthology.orchestra.run.smithi031.stdout:uid [ unknown] Ceph.com (release key) 2024-09-21T23:46:06.282 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2024-09-21T23:46:06.283 INFO:teuthology.task.install.deb:Installing system (non-project) packages: pv on remote deb x86_64 2024-09-21T23:46:06.283 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T23:46:06.297 INFO:teuthology.orchestra.run.smithi060.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-21T23:46:06.298 INFO:teuthology.orchestra.run.smithi060.stdout:uid [ unknown] Ceph.com (release key) 2024-09-21T23:46:06.298 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2024-09-21T23:46:06.298 INFO:teuthology.task.install.deb:Installing system (non-project) packages: pv on remote deb x86_64 2024-09-21T23:46:06.298 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T23:46:06.423 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default/ 2024-09-21T23:46:06.423 INFO:teuthology.task.install.deb:Package version is 19.1.1-373-g4e45e6b4-1jammy 2024-09-21T23:46:06.438 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default/ 2024-09-21T23:46:06.438 INFO:teuthology.task.install.deb:Package version is 19.1.1-373-g4e45e6b4-1jammy 2024-09-21T23:46:06.545 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:46:06.545 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-21T23:46:06.547 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:46:06.548 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-21T23:46:06.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo apt-get update 2024-09-21T23:46:06.562 DEBUG:teuthology.orchestra.run.smithi060:> sudo apt-get update 2024-09-21T23:46:06.788 INFO:teuthology.orchestra.run.smithi031.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy InRelease 2024-09-21T23:46:06.789 INFO:teuthology.orchestra.run.smithi060.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy InRelease 2024-09-21T23:46:06.809 INFO:teuthology.orchestra.run.smithi031.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-21T23:46:06.810 INFO:teuthology.orchestra.run.smithi060.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-21T23:46:06.832 INFO:teuthology.orchestra.run.smithi031.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-21T23:46:06.833 INFO:teuthology.orchestra.run.smithi060.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-21T23:46:06.855 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-21T23:46:06.863 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T23:46:06.864 INFO:teuthology.orchestra.run.smithi031.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,413 B] 2024-09-21T23:46:06.874 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T23:46:06.892 INFO:teuthology.orchestra.run.smithi031.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-21T23:46:06.895 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,413 B] 2024-09-21T23:46:06.937 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-21T23:46:06.938 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-21T23:46:06.946 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-21T23:46:06.947 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-21T23:46:07.019 INFO:teuthology.orchestra.run.smithi031.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-21T23:46:07.021 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-21T23:46:07.237 INFO:teuthology.orchestra.run.smithi031.stdout:Fetched 28.9 kB in 0s (58.0 kB/s) 2024-09-21T23:46:07.256 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 28.9 kB in 1s (57.1 kB/s) 2024-09-21T23:46:08.272 INFO:teuthology.orchestra.run.smithi031.stdout:Reading package lists... 2024-09-21T23:46:08.289 DEBUG:teuthology.orchestra.run.smithi031:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-373-g4e45e6b4-1jammy cephadm=19.1.1-373-g4e45e6b4-1jammy ceph-mds=19.1.1-373-g4e45e6b4-1jammy ceph-mgr=19.1.1-373-g4e45e6b4-1jammy ceph-common=19.1.1-373-g4e45e6b4-1jammy ceph-fuse=19.1.1-373-g4e45e6b4-1jammy ceph-test=19.1.1-373-g4e45e6b4-1jammy ceph-volume=19.1.1-373-g4e45e6b4-1jammy radosgw=19.1.1-373-g4e45e6b4-1jammy python3-rados=19.1.1-373-g4e45e6b4-1jammy python3-rgw=19.1.1-373-g4e45e6b4-1jammy python3-cephfs=19.1.1-373-g4e45e6b4-1jammy python3-rbd=19.1.1-373-g4e45e6b4-1jammy libcephfs2=19.1.1-373-g4e45e6b4-1jammy libcephfs-dev=19.1.1-373-g4e45e6b4-1jammy librados2=19.1.1-373-g4e45e6b4-1jammy librbd1=19.1.1-373-g4e45e6b4-1jammy rbd-fuse=19.1.1-373-g4e45e6b4-1jammy rbd-mirror=19.1.1-373-g4e45e6b4-1jammy 2024-09-21T23:46:08.297 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2024-09-21T23:46:08.314 DEBUG:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-373-g4e45e6b4-1jammy cephadm=19.1.1-373-g4e45e6b4-1jammy ceph-mds=19.1.1-373-g4e45e6b4-1jammy ceph-mgr=19.1.1-373-g4e45e6b4-1jammy ceph-common=19.1.1-373-g4e45e6b4-1jammy ceph-fuse=19.1.1-373-g4e45e6b4-1jammy ceph-test=19.1.1-373-g4e45e6b4-1jammy ceph-volume=19.1.1-373-g4e45e6b4-1jammy radosgw=19.1.1-373-g4e45e6b4-1jammy python3-rados=19.1.1-373-g4e45e6b4-1jammy python3-rgw=19.1.1-373-g4e45e6b4-1jammy python3-cephfs=19.1.1-373-g4e45e6b4-1jammy python3-rbd=19.1.1-373-g4e45e6b4-1jammy libcephfs2=19.1.1-373-g4e45e6b4-1jammy libcephfs-dev=19.1.1-373-g4e45e6b4-1jammy librados2=19.1.1-373-g4e45e6b4-1jammy librbd1=19.1.1-373-g4e45e6b4-1jammy rbd-fuse=19.1.1-373-g4e45e6b4-1jammy rbd-mirror=19.1.1-373-g4e45e6b4-1jammy 2024-09-21T23:46:08.344 INFO:teuthology.orchestra.run.smithi031.stdout:Reading package lists... 2024-09-21T23:46:08.369 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2024-09-21T23:46:08.535 INFO:teuthology.orchestra.run.smithi031.stdout:Building dependency tree... 2024-09-21T23:46:08.535 INFO:teuthology.orchestra.run.smithi031.stdout:Reading state information... 2024-09-21T23:46:08.567 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2024-09-21T23:46:08.568 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2024-09-21T23:46:08.729 INFO:teuthology.orchestra.run.smithi031.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T23:46:08.729 INFO:teuthology.orchestra.run.smithi031.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T23:46:08.729 INFO:teuthology.orchestra.run.smithi031.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T23:46:08.729 INFO:teuthology.orchestra.run.smithi031.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T23:46:08.730 INFO:teuthology.orchestra.run.smithi031.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T23:46:08.730 INFO:teuthology.orchestra.run.smithi031.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T23:46:08.730 INFO:teuthology.orchestra.run.smithi031.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T23:46:08.731 INFO:teuthology.orchestra.run.smithi031.stdout:The following additional packages will be installed: 2024-09-21T23:46:08.731 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-21T23:46:08.731 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-21T23:46:08.731 INFO:teuthology.orchestra.run.smithi031.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-21T23:46:08.731 INFO:teuthology.orchestra.run.smithi031.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-21T23:46:08.732 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-21T23:46:08.732 INFO:teuthology.orchestra.run.smithi031.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-21T23:46:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-21T23:46:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-21T23:46:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-21T23:46:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-21T23:46:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-21T23:46:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-21T23:46:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-21T23:46:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-21T23:46:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-21T23:46:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-21T23:46:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-21T23:46:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-21T23:46:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-21T23:46:08.733 INFO:teuthology.orchestra.run.smithi031.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-21T23:46:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-21T23:46:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-21T23:46:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-21T23:46:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet zip 2024-09-21T23:46:08.734 INFO:teuthology.orchestra.run.smithi031.stdout:Suggested packages: 2024-09-21T23:46:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-21T23:46:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-21T23:46:08.734 INFO:teuthology.orchestra.run.smithi031.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-21T23:46:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-21T23:46:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-21T23:46:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-21T23:46:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-21T23:46:08.735 INFO:teuthology.orchestra.run.smithi031.stdout:Recommended packages: 2024-09-21T23:46:08.735 INFO:teuthology.orchestra.run.smithi031.stdout: btrfs-tools 2024-09-21T23:46:08.771 INFO:teuthology.orchestra.run.smithi060.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T23:46:08.771 INFO:teuthology.orchestra.run.smithi060.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T23:46:08.771 INFO:teuthology.orchestra.run.smithi060.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T23:46:08.772 INFO:teuthology.orchestra.run.smithi060.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T23:46:08.772 INFO:teuthology.orchestra.run.smithi060.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T23:46:08.772 INFO:teuthology.orchestra.run.smithi060.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T23:46:08.772 INFO:teuthology.orchestra.run.smithi060.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T23:46:08.773 INFO:teuthology.orchestra.run.smithi060.stdout:The following additional packages will be installed: 2024-09-21T23:46:08.773 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-21T23:46:08.773 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-21T23:46:08.775 INFO:teuthology.orchestra.run.smithi060.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-21T23:46:08.775 INFO:teuthology.orchestra.run.smithi060.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-21T23:46:08.775 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-21T23:46:08.775 INFO:teuthology.orchestra.run.smithi060.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-21T23:46:08.775 INFO:teuthology.orchestra.run.smithi060.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-21T23:46:08.775 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-21T23:46:08.775 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-21T23:46:08.775 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-21T23:46:08.775 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-21T23:46:08.776 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-21T23:46:08.776 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-21T23:46:08.776 INFO:teuthology.orchestra.run.smithi060.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-21T23:46:08.776 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-21T23:46:08.776 INFO:teuthology.orchestra.run.smithi060.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-21T23:46:08.776 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-21T23:46:08.776 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-21T23:46:08.776 INFO:teuthology.orchestra.run.smithi060.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-21T23:46:08.776 INFO:teuthology.orchestra.run.smithi060.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-21T23:46:08.776 INFO:teuthology.orchestra.run.smithi060.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-21T23:46:08.776 INFO:teuthology.orchestra.run.smithi060.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-21T23:46:08.776 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-21T23:46:08.776 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet zip 2024-09-21T23:46:08.777 INFO:teuthology.orchestra.run.smithi060.stdout:Suggested packages: 2024-09-21T23:46:08.777 INFO:teuthology.orchestra.run.smithi060.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-21T23:46:08.777 INFO:teuthology.orchestra.run.smithi060.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-21T23:46:08.777 INFO:teuthology.orchestra.run.smithi060.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-21T23:46:08.777 INFO:teuthology.orchestra.run.smithi060.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-21T23:46:08.777 INFO:teuthology.orchestra.run.smithi060.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-21T23:46:08.777 INFO:teuthology.orchestra.run.smithi060.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-21T23:46:08.777 INFO:teuthology.orchestra.run.smithi060.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-21T23:46:08.777 INFO:teuthology.orchestra.run.smithi060.stdout:Recommended packages: 2024-09-21T23:46:08.778 INFO:teuthology.orchestra.run.smithi060.stdout: btrfs-tools 2024-09-21T23:46:08.840 INFO:teuthology.orchestra.run.smithi031.stdout:The following NEW packages will be installed: 2024-09-21T23:46:08.841 INFO:teuthology.orchestra.run.smithi031.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-21T23:46:08.841 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-21T23:46:08.841 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-21T23:46:08.841 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-21T23:46:08.841 INFO:teuthology.orchestra.run.smithi031.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-21T23:46:08.841 INFO:teuthology.orchestra.run.smithi031.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-21T23:46:08.842 INFO:teuthology.orchestra.run.smithi031.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-21T23:46:08.842 INFO:teuthology.orchestra.run.smithi031.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-21T23:46:08.842 INFO:teuthology.orchestra.run.smithi031.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-21T23:46:08.842 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-21T23:46:08.843 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-21T23:46:08.843 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-21T23:46:08.843 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-21T23:46:08.843 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-21T23:46:08.843 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-21T23:46:08.843 INFO:teuthology.orchestra.run.smithi031.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-21T23:46:08.843 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-21T23:46:08.843 INFO:teuthology.orchestra.run.smithi031.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-21T23:46:08.843 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-21T23:46:08.843 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-21T23:46:08.843 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-21T23:46:08.843 INFO:teuthology.orchestra.run.smithi031.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-21T23:46:08.843 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-21T23:46:08.844 INFO:teuthology.orchestra.run.smithi031.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-21T23:46:08.844 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-21T23:46:08.844 INFO:teuthology.orchestra.run.smithi031.stdout: radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip 2024-09-21T23:46:08.884 INFO:teuthology.orchestra.run.smithi060.stdout:The following NEW packages will be installed: 2024-09-21T23:46:08.884 INFO:teuthology.orchestra.run.smithi060.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-21T23:46:08.884 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-21T23:46:08.884 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-21T23:46:08.884 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-21T23:46:08.884 INFO:teuthology.orchestra.run.smithi060.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-21T23:46:08.884 INFO:teuthology.orchestra.run.smithi060.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-21T23:46:08.886 INFO:teuthology.orchestra.run.smithi060.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-21T23:46:08.886 INFO:teuthology.orchestra.run.smithi060.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-21T23:46:08.886 INFO:teuthology.orchestra.run.smithi060.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-21T23:46:08.886 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-21T23:46:08.886 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-21T23:46:08.886 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-21T23:46:08.886 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-21T23:46:08.886 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-21T23:46:08.886 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-21T23:46:08.886 INFO:teuthology.orchestra.run.smithi060.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-21T23:46:08.886 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-21T23:46:08.886 INFO:teuthology.orchestra.run.smithi060.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-21T23:46:08.887 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-21T23:46:08.887 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-21T23:46:08.887 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-21T23:46:08.887 INFO:teuthology.orchestra.run.smithi060.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-21T23:46:08.887 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-21T23:46:08.887 INFO:teuthology.orchestra.run.smithi060.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-21T23:46:08.887 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-21T23:46:08.887 INFO:teuthology.orchestra.run.smithi060.stdout: radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip 2024-09-21T23:46:08.905 INFO:teuthology.orchestra.run.smithi031.stdout:0 upgraded, 119 newly installed, 0 to remove and 332 not upgraded. 2024-09-21T23:46:08.905 INFO:teuthology.orchestra.run.smithi031.stdout:Need to get 184 MB of archives. 2024-09-21T23:46:08.905 INFO:teuthology.orchestra.run.smithi031.stdout:After this operation, 842 MB of additional disk space will be used. 2024-09-21T23:46:08.905 INFO:teuthology.orchestra.run.smithi031.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-21T23:46:08.989 INFO:teuthology.orchestra.run.smithi031.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-373-g4e45e6b4-1jammy [3,592 kB] 2024-09-21T23:46:08.995 INFO:teuthology.orchestra.run.smithi031.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-21T23:46:08.997 INFO:teuthology.orchestra.run.smithi031.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-21T23:46:09.015 INFO:teuthology.orchestra.run.smithi031.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-21T23:46:09.033 INFO:teuthology.orchestra.run.smithi060.stdout:0 upgraded, 119 newly installed, 0 to remove and 332 not upgraded. 2024-09-21T23:46:09.033 INFO:teuthology.orchestra.run.smithi060.stdout:Need to get 184 MB of archives. 2024-09-21T23:46:09.033 INFO:teuthology.orchestra.run.smithi060.stdout:After this operation, 842 MB of additional disk space will be used. 2024-09-21T23:46:09.033 INFO:teuthology.orchestra.run.smithi060.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-373-g4e45e6b4-1jammy [3,592 kB] 2024-09-21T23:46:09.075 INFO:teuthology.orchestra.run.smithi031.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-21T23:46:09.078 INFO:teuthology.orchestra.run.smithi060.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-21T23:46:09.079 INFO:teuthology.orchestra.run.smithi031.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-21T23:46:09.092 INFO:teuthology.orchestra.run.smithi031.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-21T23:46:09.096 INFO:teuthology.orchestra.run.smithi031.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-21T23:46:09.097 INFO:teuthology.orchestra.run.smithi031.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-21T23:46:09.098 INFO:teuthology.orchestra.run.smithi031.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-21T23:46:09.107 INFO:teuthology.orchestra.run.smithi031.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-21T23:46:09.109 INFO:teuthology.orchestra.run.smithi031.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-21T23:46:09.112 INFO:teuthology.orchestra.run.smithi031.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-21T23:46:09.113 INFO:teuthology.orchestra.run.smithi031.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-21T23:46:09.114 INFO:teuthology.orchestra.run.smithi031.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-21T23:46:09.116 INFO:teuthology.orchestra.run.smithi031.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-21T23:46:09.131 INFO:teuthology.orchestra.run.smithi031.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-21T23:46:09.133 INFO:teuthology.orchestra.run.smithi031.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-21T23:46:09.134 INFO:teuthology.orchestra.run.smithi031.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-21T23:46:09.134 INFO:teuthology.orchestra.run.smithi031.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-21T23:46:09.135 INFO:teuthology.orchestra.run.smithi031.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-21T23:46:09.135 INFO:teuthology.orchestra.run.smithi031.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-21T23:46:09.148 INFO:teuthology.orchestra.run.smithi031.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-21T23:46:09.148 INFO:teuthology.orchestra.run.smithi031.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-21T23:46:09.149 INFO:teuthology.orchestra.run.smithi031.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-21T23:46:09.149 INFO:teuthology.orchestra.run.smithi031.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-21T23:46:09.165 INFO:teuthology.orchestra.run.smithi031.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-21T23:46:09.168 INFO:teuthology.orchestra.run.smithi031.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-21T23:46:09.168 INFO:teuthology.orchestra.run.smithi031.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-21T23:46:09.169 INFO:teuthology.orchestra.run.smithi031.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-21T23:46:09.169 INFO:teuthology.orchestra.run.smithi031.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-21T23:46:09.170 INFO:teuthology.orchestra.run.smithi031.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-21T23:46:09.173 INFO:teuthology.orchestra.run.smithi031.stdout:Get:34 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-373-g4e45e6b4-1jammy [3,241 kB] 2024-09-21T23:46:09.183 INFO:teuthology.orchestra.run.smithi031.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-21T23:46:09.183 INFO:teuthology.orchestra.run.smithi031.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-21T23:46:09.184 INFO:teuthology.orchestra.run.smithi031.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-21T23:46:09.185 INFO:teuthology.orchestra.run.smithi031.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-21T23:46:09.200 INFO:teuthology.orchestra.run.smithi031.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-21T23:46:09.206 INFO:teuthology.orchestra.run.smithi031.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-21T23:46:09.206 INFO:teuthology.orchestra.run.smithi031.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-21T23:46:09.206 INFO:teuthology.orchestra.run.smithi031.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-21T23:46:09.207 INFO:teuthology.orchestra.run.smithi031.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-21T23:46:09.209 INFO:teuthology.orchestra.run.smithi031.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-21T23:46:09.217 INFO:teuthology.orchestra.run.smithi031.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-21T23:46:09.218 INFO:teuthology.orchestra.run.smithi031.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-21T23:46:09.220 INFO:teuthology.orchestra.run.smithi031.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-21T23:46:09.220 INFO:teuthology.orchestra.run.smithi031.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-21T23:46:09.233 INFO:teuthology.orchestra.run.smithi031.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-373-g4e45e6b4-1jammy [719 kB] 2024-09-21T23:46:09.234 INFO:teuthology.orchestra.run.smithi031.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-21T23:46:09.235 INFO:teuthology.orchestra.run.smithi031.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-21T23:46:09.236 INFO:teuthology.orchestra.run.smithi031.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-21T23:46:09.239 INFO:teuthology.orchestra.run.smithi031.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-21T23:46:09.249 INFO:teuthology.orchestra.run.smithi031.stdout:Get:54 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-373-g4e45e6b4-1jammy [356 kB] 2024-09-21T23:46:09.256 INFO:teuthology.orchestra.run.smithi031.stdout:Get:55 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-373-g4e45e6b4-1jammy [32.8 kB] 2024-09-21T23:46:09.257 INFO:teuthology.orchestra.run.smithi031.stdout:Get:56 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-373-g4e45e6b4-1jammy [180 kB] 2024-09-21T23:46:09.261 INFO:teuthology.orchestra.run.smithi031.stdout:Get:57 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-373-g4e45e6b4-1jammy [67.8 kB] 2024-09-21T23:46:09.262 INFO:teuthology.orchestra.run.smithi031.stdout:Get:58 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-373-g4e45e6b4-1jammy [333 kB] 2024-09-21T23:46:09.268 INFO:teuthology.orchestra.run.smithi031.stdout:Get:59 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-373-g4e45e6b4-1jammy [6,875 kB] 2024-09-21T23:46:09.292 INFO:teuthology.orchestra.run.smithi031.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-21T23:46:09.293 INFO:teuthology.orchestra.run.smithi031.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-21T23:46:09.294 INFO:teuthology.orchestra.run.smithi031.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-21T23:46:09.295 INFO:teuthology.orchestra.run.smithi031.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-21T23:46:09.296 INFO:teuthology.orchestra.run.smithi031.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-21T23:46:09.313 INFO:teuthology.orchestra.run.smithi031.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-21T23:46:09.315 INFO:teuthology.orchestra.run.smithi031.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-21T23:46:09.315 INFO:teuthology.orchestra.run.smithi031.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-21T23:46:09.330 INFO:teuthology.orchestra.run.smithi031.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-21T23:46:09.330 INFO:teuthology.orchestra.run.smithi031.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-21T23:46:09.331 INFO:teuthology.orchestra.run.smithi031.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-21T23:46:09.331 INFO:teuthology.orchestra.run.smithi031.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-21T23:46:09.332 INFO:teuthology.orchestra.run.smithi031.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-21T23:46:09.332 INFO:teuthology.orchestra.run.smithi031.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-21T23:46:09.345 INFO:teuthology.orchestra.run.smithi031.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-21T23:46:09.346 INFO:teuthology.orchestra.run.smithi031.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-21T23:46:09.349 INFO:teuthology.orchestra.run.smithi031.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-21T23:46:09.356 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-373-g4e45e6b4-1jammy [3,241 kB] 2024-09-21T23:46:09.362 INFO:teuthology.orchestra.run.smithi031.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-21T23:46:09.364 INFO:teuthology.orchestra.run.smithi031.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-21T23:46:09.364 INFO:teuthology.orchestra.run.smithi031.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-21T23:46:09.367 INFO:teuthology.orchestra.run.smithi031.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-21T23:46:09.370 INFO:teuthology.orchestra.run.smithi031.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-21T23:46:09.370 INFO:teuthology.orchestra.run.smithi031.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-21T23:46:09.370 INFO:teuthology.orchestra.run.smithi031.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-21T23:46:09.379 INFO:teuthology.orchestra.run.smithi031.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-21T23:46:09.380 INFO:teuthology.orchestra.run.smithi031.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-21T23:46:09.383 INFO:teuthology.orchestra.run.smithi031.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-21T23:46:09.397 INFO:teuthology.orchestra.run.smithi031.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-21T23:46:09.398 INFO:teuthology.orchestra.run.smithi031.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-21T23:46:09.398 INFO:teuthology.orchestra.run.smithi031.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-21T23:46:09.399 INFO:teuthology.orchestra.run.smithi031.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-21T23:46:09.400 INFO:teuthology.orchestra.run.smithi031.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-21T23:46:09.406 INFO:teuthology.orchestra.run.smithi031.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-21T23:46:09.406 INFO:teuthology.orchestra.run.smithi031.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-21T23:46:09.414 INFO:teuthology.orchestra.run.smithi031.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-21T23:46:09.416 INFO:teuthology.orchestra.run.smithi031.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-21T23:46:09.417 INFO:teuthology.orchestra.run.smithi031.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-21T23:46:09.418 INFO:teuthology.orchestra.run.smithi031.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-373-g4e45e6b4-1jammy [112 kB] 2024-09-21T23:46:09.421 INFO:teuthology.orchestra.run.smithi031.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-373-g4e45e6b4-1jammy [472 kB] 2024-09-21T23:46:09.427 INFO:teuthology.orchestra.run.smithi031.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-373-g4e45e6b4-1jammy [25.9 MB] 2024-09-21T23:46:09.501 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-21T23:46:09.513 INFO:teuthology.orchestra.run.smithi060.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-21T23:46:09.565 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-373-g4e45e6b4-1jammy [719 kB] 2024-09-21T23:46:09.594 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-21T23:46:09.619 INFO:teuthology.orchestra.run.smithi060.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-373-g4e45e6b4-1jammy [356 kB] 2024-09-21T23:46:09.667 INFO:teuthology.orchestra.run.smithi060.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-373-g4e45e6b4-1jammy [32.8 kB] 2024-09-21T23:46:09.669 INFO:teuthology.orchestra.run.smithi060.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-373-g4e45e6b4-1jammy [180 kB] 2024-09-21T23:46:09.673 INFO:teuthology.orchestra.run.smithi060.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-373-g4e45e6b4-1jammy [67.8 kB] 2024-09-21T23:46:09.674 INFO:teuthology.orchestra.run.smithi060.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-373-g4e45e6b4-1jammy [333 kB] 2024-09-21T23:46:09.681 INFO:teuthology.orchestra.run.smithi060.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-373-g4e45e6b4-1jammy [6,875 kB] 2024-09-21T23:46:09.824 INFO:teuthology.orchestra.run.smithi060.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-21T23:46:09.835 INFO:teuthology.orchestra.run.smithi060.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-21T23:46:09.865 INFO:teuthology.orchestra.run.smithi060.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-21T23:46:09.873 INFO:teuthology.orchestra.run.smithi060.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-21T23:46:09.874 INFO:teuthology.orchestra.run.smithi060.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-21T23:46:09.876 INFO:teuthology.orchestra.run.smithi060.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-21T23:46:09.893 INFO:teuthology.orchestra.run.smithi060.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-21T23:46:09.898 INFO:teuthology.orchestra.run.smithi060.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-21T23:46:09.902 INFO:teuthology.orchestra.run.smithi060.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-21T23:46:09.906 INFO:teuthology.orchestra.run.smithi060.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-21T23:46:09.987 INFO:teuthology.orchestra.run.smithi060.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-21T23:46:09.992 INFO:teuthology.orchestra.run.smithi060.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-21T23:46:09.996 INFO:teuthology.orchestra.run.smithi060.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-21T23:46:10.000 INFO:teuthology.orchestra.run.smithi060.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-21T23:46:10.002 INFO:teuthology.orchestra.run.smithi060.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-21T23:46:10.003 INFO:teuthology.orchestra.run.smithi060.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-21T23:46:10.005 INFO:teuthology.orchestra.run.smithi060.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-21T23:46:10.006 INFO:teuthology.orchestra.run.smithi060.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-21T23:46:10.007 INFO:teuthology.orchestra.run.smithi060.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-21T23:46:10.050 INFO:teuthology.orchestra.run.smithi031.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-373-g4e45e6b4-1jammy [5,145 kB] 2024-09-21T23:46:10.070 INFO:teuthology.orchestra.run.smithi060.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-21T23:46:10.090 INFO:teuthology.orchestra.run.smithi060.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-21T23:46:10.091 INFO:teuthology.orchestra.run.smithi060.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-21T23:46:10.091 INFO:teuthology.orchestra.run.smithi060.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-21T23:46:10.093 INFO:teuthology.orchestra.run.smithi060.stdout:Get:37 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-373-g4e45e6b4-1jammy [112 kB] 2024-09-21T23:46:10.095 INFO:teuthology.orchestra.run.smithi060.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-21T23:46:10.096 INFO:teuthology.orchestra.run.smithi060.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-21T23:46:10.096 INFO:teuthology.orchestra.run.smithi060.stdout:Get:40 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-373-g4e45e6b4-1jammy [472 kB] 2024-09-21T23:46:10.117 INFO:teuthology.orchestra.run.smithi031.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-373-g4e45e6b4-1jammy [245 kB] 2024-09-21T23:46:10.121 INFO:teuthology.orchestra.run.smithi060.stdout:Get:41 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-373-g4e45e6b4-1jammy [25.9 MB] 2024-09-21T23:46:10.126 INFO:teuthology.orchestra.run.smithi031.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-373-g4e45e6b4-1jammy [124 kB] 2024-09-21T23:46:10.128 INFO:teuthology.orchestra.run.smithi031.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-373-g4e45e6b4-1jammy [1,555 kB] 2024-09-21T23:46:10.152 INFO:teuthology.orchestra.run.smithi060.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-21T23:46:10.153 INFO:teuthology.orchestra.run.smithi060.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-21T23:46:10.155 INFO:teuthology.orchestra.run.smithi060.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-21T23:46:10.155 INFO:teuthology.orchestra.run.smithi060.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-21T23:46:10.163 INFO:teuthology.orchestra.run.smithi031.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-373-g4e45e6b4-1jammy [6,142 kB] 2024-09-21T23:46:10.233 INFO:teuthology.orchestra.run.smithi060.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-21T23:46:10.235 INFO:teuthology.orchestra.run.smithi060.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-21T23:46:10.237 INFO:teuthology.orchestra.run.smithi060.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-21T23:46:10.238 INFO:teuthology.orchestra.run.smithi060.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-21T23:46:10.249 INFO:teuthology.orchestra.run.smithi060.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-21T23:46:10.309 INFO:teuthology.orchestra.run.smithi031.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-373-g4e45e6b4-1jammy [22.8 MB] 2024-09-21T23:46:10.314 INFO:teuthology.orchestra.run.smithi060.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-21T23:46:10.316 INFO:teuthology.orchestra.run.smithi060.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-21T23:46:10.317 INFO:teuthology.orchestra.run.smithi060.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-21T23:46:10.319 INFO:teuthology.orchestra.run.smithi060.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-21T23:46:10.324 INFO:teuthology.orchestra.run.smithi060.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-21T23:46:10.396 INFO:teuthology.orchestra.run.smithi060.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-21T23:46:10.399 INFO:teuthology.orchestra.run.smithi060.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-21T23:46:10.401 INFO:teuthology.orchestra.run.smithi060.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-21T23:46:10.406 INFO:teuthology.orchestra.run.smithi060.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-21T23:46:10.407 INFO:teuthology.orchestra.run.smithi060.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-21T23:46:10.478 INFO:teuthology.orchestra.run.smithi060.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-21T23:46:10.482 INFO:teuthology.orchestra.run.smithi060.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-21T23:46:10.647 INFO:teuthology.orchestra.run.smithi060.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-21T23:46:10.648 INFO:teuthology.orchestra.run.smithi060.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-21T23:46:10.651 INFO:teuthology.orchestra.run.smithi060.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-21T23:46:10.652 INFO:teuthology.orchestra.run.smithi060.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-21T23:46:10.654 INFO:teuthology.orchestra.run.smithi060.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-21T23:46:10.691 INFO:teuthology.orchestra.run.smithi060.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-21T23:46:10.694 INFO:teuthology.orchestra.run.smithi060.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-21T23:46:10.695 INFO:teuthology.orchestra.run.smithi060.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-21T23:46:10.751 INFO:teuthology.orchestra.run.smithi060.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-21T23:46:10.751 INFO:teuthology.orchestra.run.smithi060.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-21T23:46:10.813 INFO:teuthology.orchestra.run.smithi060.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-21T23:46:10.814 INFO:teuthology.orchestra.run.smithi060.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-21T23:46:10.816 INFO:teuthology.orchestra.run.smithi060.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-21T23:46:10.817 INFO:teuthology.orchestra.run.smithi060.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-21T23:46:10.818 INFO:teuthology.orchestra.run.smithi060.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-21T23:46:10.819 INFO:teuthology.orchestra.run.smithi060.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-21T23:46:10.826 INFO:teuthology.orchestra.run.smithi060.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-21T23:46:10.895 INFO:teuthology.orchestra.run.smithi060.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-21T23:46:10.899 INFO:teuthology.orchestra.run.smithi060.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-21T23:46:10.901 INFO:teuthology.orchestra.run.smithi060.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-21T23:46:10.977 INFO:teuthology.orchestra.run.smithi060.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-21T23:46:10.984 INFO:teuthology.orchestra.run.smithi060.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-21T23:46:10.986 INFO:teuthology.orchestra.run.smithi060.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-21T23:46:10.988 INFO:teuthology.orchestra.run.smithi060.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-21T23:46:10.999 INFO:teuthology.orchestra.run.smithi060.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-21T23:46:11.001 INFO:teuthology.orchestra.run.smithi060.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-21T23:46:11.009 INFO:teuthology.orchestra.run.smithi060.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-21T23:46:11.011 INFO:teuthology.orchestra.run.smithi031.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-373-g4e45e6b4-1jammy [14.1 kB] 2024-09-21T23:46:11.011 INFO:teuthology.orchestra.run.smithi031.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-373-g4e45e6b4-1jammy [877 kB] 2024-09-21T23:46:11.023 INFO:teuthology.orchestra.run.smithi031.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-373-g4e45e6b4-1jammy [2,462 kB] 2024-09-21T23:46:11.054 INFO:teuthology.orchestra.run.smithi031.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-373-g4e45e6b4-1jammy [794 kB] 2024-09-21T23:46:11.058 INFO:teuthology.orchestra.run.smithi060.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-21T23:46:11.060 INFO:teuthology.orchestra.run.smithi060.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-21T23:46:11.061 INFO:teuthology.orchestra.run.smithi060.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-21T23:46:11.065 INFO:teuthology.orchestra.run.smithi031.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-373-g4e45e6b4-1jammy [152 kB] 2024-09-21T23:46:11.067 INFO:teuthology.orchestra.run.smithi031.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-373-g4e45e6b4-1jammy [2,347 kB] 2024-09-21T23:46:11.129 INFO:teuthology.orchestra.run.smithi031.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-373-g4e45e6b4-1jammy [8,625 kB] 2024-09-21T23:46:11.140 INFO:teuthology.orchestra.run.smithi060.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-21T23:46:11.142 INFO:teuthology.orchestra.run.smithi060.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-21T23:46:11.160 INFO:teuthology.orchestra.run.smithi060.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-21T23:46:11.161 INFO:teuthology.orchestra.run.smithi060.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-21T23:46:11.162 INFO:teuthology.orchestra.run.smithi060.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-21T23:46:11.167 INFO:teuthology.orchestra.run.smithi060.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-21T23:46:11.169 INFO:teuthology.orchestra.run.smithi060.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-21T23:46:11.339 INFO:teuthology.orchestra.run.smithi060.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-373-g4e45e6b4-1jammy [5,145 kB] 2024-09-21T23:46:11.350 INFO:teuthology.orchestra.run.smithi031.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-373-g4e45e6b4-1jammy [14.2 kB] 2024-09-21T23:46:11.350 INFO:teuthology.orchestra.run.smithi031.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-373-g4e45e6b4-1jammy [51.4 MB] 2024-09-21T23:46:11.436 INFO:teuthology.orchestra.run.smithi060.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-373-g4e45e6b4-1jammy [245 kB] 2024-09-21T23:46:11.443 INFO:teuthology.orchestra.run.smithi060.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-373-g4e45e6b4-1jammy [124 kB] 2024-09-21T23:46:11.452 INFO:teuthology.orchestra.run.smithi060.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-373-g4e45e6b4-1jammy [1,555 kB] 2024-09-21T23:46:11.508 INFO:teuthology.orchestra.run.smithi060.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-373-g4e45e6b4-1jammy [6,142 kB] 2024-09-21T23:46:11.731 INFO:teuthology.orchestra.run.smithi060.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-373-g4e45e6b4-1jammy [22.8 MB] 2024-09-21T23:46:12.583 INFO:teuthology.orchestra.run.smithi060.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-373-g4e45e6b4-1jammy [14.1 kB] 2024-09-21T23:46:12.584 INFO:teuthology.orchestra.run.smithi060.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-373-g4e45e6b4-1jammy [877 kB] 2024-09-21T23:46:12.601 INFO:teuthology.orchestra.run.smithi060.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-373-g4e45e6b4-1jammy [2,462 kB] 2024-09-21T23:46:12.644 INFO:teuthology.orchestra.run.smithi060.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-373-g4e45e6b4-1jammy [794 kB] 2024-09-21T23:46:12.654 INFO:teuthology.orchestra.run.smithi060.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-373-g4e45e6b4-1jammy [152 kB] 2024-09-21T23:46:12.656 INFO:teuthology.orchestra.run.smithi060.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-373-g4e45e6b4-1jammy [2,347 kB] 2024-09-21T23:46:12.716 INFO:teuthology.orchestra.run.smithi060.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-373-g4e45e6b4-1jammy [8,625 kB] 2024-09-21T23:46:12.983 INFO:teuthology.orchestra.run.smithi060.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-373-g4e45e6b4-1jammy [14.2 kB] 2024-09-21T23:46:12.984 INFO:teuthology.orchestra.run.smithi060.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-373-g4e45e6b4-1jammy [51.4 MB] 2024-09-21T23:46:13.232 INFO:teuthology.orchestra.run.smithi031.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-373-g4e45e6b4-1jammy [130 kB] 2024-09-21T23:46:13.235 INFO:teuthology.orchestra.run.smithi031.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-373-g4e45e6b4-1jammy [39.0 kB] 2024-09-21T23:46:13.236 INFO:teuthology.orchestra.run.smithi031.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-373-g4e45e6b4-1jammy [13.6 MB] 2024-09-21T23:46:13.610 INFO:teuthology.orchestra.run.smithi031.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-373-g4e45e6b4-1jammy [91.4 kB] 2024-09-21T23:46:13.612 INFO:teuthology.orchestra.run.smithi031.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.1.1-373-g4e45e6b4-1jammy [3,140 kB] 2024-09-21T23:46:14.305 INFO:teuthology.orchestra.run.smithi031.stdout:Fetched 184 MB in 5s (37.9 MB/s) 2024-09-21T23:46:14.341 INFO:teuthology.orchestra.run.smithi060.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-373-g4e45e6b4-1jammy [130 kB] 2024-09-21T23:46:14.344 INFO:teuthology.orchestra.run.smithi060.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-373-g4e45e6b4-1jammy [39.0 kB] 2024-09-21T23:46:14.345 INFO:teuthology.orchestra.run.smithi060.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-373-g4e45e6b4-1jammy [13.6 MB] 2024-09-21T23:46:14.478 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-21T23:46:14.534 INFO:teuthology.orchestra.run.smithi060.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-373-g4e45e6b4-1jammy [91.4 kB] 2024-09-21T23:46:14.536 INFO:teuthology.orchestra.run.smithi060.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.1.1-373-g4e45e6b4-1jammy [3,140 kB] 2024-09-21T23:46:14.581 INFO:teuthology.orchestra.run.smithi031.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-21T23:46:14.587 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-21T23:46:14.668 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-21T23:46:14.988 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-21T23:46:15.015 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-21T23:46:15.055 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-21T23:46:15.180 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 184 MB in 6s (32.4 MB/s) 2024-09-21T23:46:15.309 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-21T23:46:15.391 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-21T23:46:15.410 INFO:teuthology.orchestra.run.smithi060.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-21T23:46:15.416 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-21T23:46:15.418 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-21T23:46:15.457 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-21T23:46:15.482 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-21T23:46:15.769 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-21T23:46:15.796 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-21T23:46:15.827 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-21T23:46:15.844 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-21T23:46:15.871 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T23:46:15.911 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T23:46:16.164 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-21T23:46:16.191 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-21T23:46:16.230 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-21T23:46:16.263 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-21T23:46:16.279 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T23:46:16.313 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T23:46:16.666 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-21T23:46:16.684 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-21T23:46:16.693 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T23:46:16.711 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T23:46:16.733 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T23:46:16.759 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T23:46:17.094 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-21T23:46:17.121 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-21T23:46:17.177 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-21T23:46:17.198 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-21T23:46:17.225 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T23:46:17.264 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T23:46:17.464 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package librados2. 2024-09-21T23:46:17.490 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../007-librados2_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:17.521 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking librados2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:17.592 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-21T23:46:17.619 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T23:46:17.659 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T23:46:17.966 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package librbd1. 2024-09-21T23:46:17.978 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-21T23:46:17.992 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../008-librbd1_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:17.995 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-21T23:46:18.028 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-21T23:46:18.032 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking librbd1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:18.365 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librados2. 2024-09-21T23:46:18.392 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../007-librados2_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:18.432 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librados2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:18.469 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libcephfs2. 2024-09-21T23:46:18.495 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:18.535 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libcephfs2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:18.829 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-rados. 2024-09-21T23:46:18.845 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../010-python3-rados_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:18.868 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librbd1. 2024-09-21T23:46:18.887 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-rados (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:18.895 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../008-librbd1_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:18.934 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librbd1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:19.182 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-21T23:46:19.198 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T23:46:19.231 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-ceph-argparse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:19.346 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libcephfs2. 2024-09-21T23:46:19.373 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:19.404 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libcephfs2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:19.459 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-cephfs. 2024-09-21T23:46:19.475 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:19.517 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-cephfs (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:19.690 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rados. 2024-09-21T23:46:19.706 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../010-python3-rados_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:19.748 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rados (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:19.820 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-21T23:46:19.846 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T23:46:19.886 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:20.076 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-21T23:46:20.103 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T23:46:20.143 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-ceph-argparse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:20.181 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-21T23:46:20.197 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-21T23:46:20.239 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-21T23:46:20.395 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cephfs. 2024-09-21T23:46:20.422 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:20.462 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cephfs (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:20.558 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-prettytable. 2024-09-21T23:46:20.585 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-21T23:46:20.625 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-21T23:46:20.731 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-21T23:46:20.747 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T23:46:20.781 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:20.903 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-rbd. 2024-09-21T23:46:20.929 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:20.969 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-rbd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:21.068 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-21T23:46:21.095 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-21T23:46:21.135 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-21T23:46:21.280 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-21T23:46:21.307 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-21T23:46:21.338 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-21T23:46:21.429 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-prettytable. 2024-09-21T23:46:21.456 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-21T23:46:21.495 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-21T23:46:21.574 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-21T23:46:21.590 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-21T23:46:21.624 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-21T23:46:21.926 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-21T23:46:21.953 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-21T23:46:21.965 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rbd. 2024-09-21T23:46:21.993 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:22.001 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-21T23:46:22.032 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rbd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:22.338 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package lua5.1. 2024-09-21T23:46:22.365 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-21T23:46:22.385 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-21T23:46:22.404 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-21T23:46:22.412 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-21T23:46:22.451 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-21T23:46:22.699 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package lua-any. 2024-09-21T23:46:22.715 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-21T23:46:22.721 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-21T23:46:22.747 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-21T23:46:22.774 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-21T23:46:22.787 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-21T23:46:22.959 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package zip. 2024-09-21T23:46:22.983 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-21T23:46:23.026 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking zip (3.0-12build2) ... 2024-09-21T23:46:23.057 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-21T23:46:23.084 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-21T23:46:23.124 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-21T23:46:23.363 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package unzip. 2024-09-21T23:46:23.389 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-21T23:46:23.429 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-21T23:46:23.452 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package lua5.1. 2024-09-21T23:46:23.479 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-21T23:46:23.518 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-21T23:46:23.839 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package lua-any. 2024-09-21T23:46:23.865 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-21T23:46:23.905 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-21T23:46:23.916 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package luarocks. 2024-09-21T23:46:23.943 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-21T23:46:23.982 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-21T23:46:24.133 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package zip. 2024-09-21T23:46:24.159 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-21T23:46:24.199 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking zip (3.0-12build2) ... 2024-09-21T23:46:24.348 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package librgw2. 2024-09-21T23:46:24.375 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../025-librgw2_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:24.414 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking librgw2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:24.511 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package unzip. 2024-09-21T23:46:24.537 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-21T23:46:24.577 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-21T23:46:24.933 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-rgw. 2024-09-21T23:46:24.949 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:24.983 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-rgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:24.990 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package luarocks. 2024-09-21T23:46:25.017 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-21T23:46:25.056 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-21T23:46:25.361 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-21T23:46:25.388 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-21T23:46:25.428 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-21T23:46:25.448 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librgw2. 2024-09-21T23:46:25.475 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../025-librgw2_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:25.523 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librgw2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:25.739 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libradosstriper1. 2024-09-21T23:46:25.765 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:25.805 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libradosstriper1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:26.092 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rgw. 2024-09-21T23:46:26.108 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:26.142 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:26.217 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-common. 2024-09-21T23:46:26.243 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../029-ceph-common_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:26.283 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:26.470 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-21T23:46:26.497 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-21T23:46:26.536 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-21T23:46:26.873 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libradosstriper1. 2024-09-21T23:46:26.900 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:26.940 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libradosstriper1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:27.318 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-common. 2024-09-21T23:46:27.346 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../029-ceph-common_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:27.385 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:27.973 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-base. 2024-09-21T23:46:27.998 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../030-ceph-base_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:28.048 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-base (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:28.634 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-21T23:46:28.660 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-21T23:46:28.717 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-21T23:46:28.832 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-base. 2024-09-21T23:46:28.859 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../030-ceph-base_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:28.908 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-base (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:29.095 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-cheroot. 2024-09-21T23:46:29.121 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-21T23:46:29.170 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-21T23:46:29.445 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-21T23:46:29.461 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-21T23:46:29.494 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-21T23:46:29.556 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-21T23:46:29.583 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-21T23:46:29.631 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-21T23:46:29.806 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cheroot. 2024-09-21T23:46:29.833 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-21T23:46:29.880 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-21T23:46:29.925 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-21T23:46:29.952 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-21T23:46:29.991 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-21T23:46:30.209 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-21T23:46:30.236 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-21T23:46:30.275 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-21T23:46:30.336 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-21T23:46:30.363 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-21T23:46:30.411 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-21T23:46:30.579 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-21T23:46:30.605 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-21T23:46:30.645 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-21T23:46:30.730 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-tempora. 2024-09-21T23:46:30.757 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-21T23:46:30.797 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-21T23:46:31.015 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-21T23:46:31.042 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-21T23:46:31.074 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-portend. 2024-09-21T23:46:31.090 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-21T23:46:31.101 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-21T23:46:31.132 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-21T23:46:31.393 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-tempora. 2024-09-21T23:46:31.420 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-21T23:46:31.427 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-21T23:46:31.454 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-21T23:46:31.459 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-21T23:46:31.494 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-21T23:46:31.762 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-portend. 2024-09-21T23:46:31.789 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-21T23:46:31.796 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-21T23:46:31.823 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-21T23:46:31.829 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-21T23:46:31.863 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-21T23:46:32.099 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-21T23:46:32.125 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-21T23:46:32.224 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-natsort. 2024-09-21T23:46:32.251 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-21T23:46:32.291 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-21T23:46:32.323 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-21T23:46:32.579 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-logutils. 2024-09-21T23:46:32.604 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-21T23:46:32.626 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-21T23:46:32.646 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-21T23:46:32.654 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-21T23:46:32.693 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-21T23:46:32.949 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-21T23:46:32.976 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-21T23:46:33.015 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-21T23:46:33.022 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-natsort. 2024-09-21T23:46:33.035 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-21T23:46:33.063 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-21T23:46:33.318 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-mako. 2024-09-21T23:46:33.345 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-21T23:46:33.350 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-logutils. 2024-09-21T23:46:33.377 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-21T23:46:33.384 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-21T23:46:33.450 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-21T23:46:33.753 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-21T23:46:33.780 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-21T23:46:33.788 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-21T23:46:33.815 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-21T23:46:33.819 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-21T23:46:33.863 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-21T23:46:34.122 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-mako. 2024-09-21T23:46:34.149 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-21T23:46:34.188 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-21T23:46:34.232 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-21T23:46:34.259 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-21T23:46:34.299 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-21T23:46:34.484 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-21T23:46:34.511 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-21T23:46:34.542 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-21T23:46:34.602 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-webob. 2024-09-21T23:46:34.629 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-21T23:46:34.668 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-21T23:46:34.811 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-21T23:46:34.839 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-21T23:46:34.878 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-21T23:46:35.022 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-waitress. 2024-09-21T23:46:35.048 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-21T23:46:35.091 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-21T23:46:35.164 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-webob. 2024-09-21T23:46:35.191 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-21T23:46:35.231 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-21T23:46:35.375 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-tempita. 2024-09-21T23:46:35.401 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-21T23:46:35.441 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-21T23:46:35.601 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-waitress. 2024-09-21T23:46:35.628 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-21T23:46:35.670 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-21T23:46:35.744 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-paste. 2024-09-21T23:46:35.771 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-21T23:46:35.810 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-21T23:46:35.987 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-tempita. 2024-09-21T23:46:36.014 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-21T23:46:36.053 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-21T23:46:36.097 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-21T23:46:36.124 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-21T23:46:36.163 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-21T23:46:36.306 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-paste. 2024-09-21T23:46:36.322 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-21T23:46:36.357 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-21T23:46:36.466 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-21T23:46:36.493 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-21T23:46:36.533 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-21T23:46:36.693 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-21T23:46:36.709 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-21T23:46:36.759 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-21T23:46:36.836 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-webtest. 2024-09-21T23:46:36.863 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-21T23:46:36.902 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-21T23:46:37.138 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-21T23:46:37.165 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-21T23:46:37.197 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pecan. 2024-09-21T23:46:37.204 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-21T23:46:37.224 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-21T23:46:37.264 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-21T23:46:37.541 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-webtest. 2024-09-21T23:46:37.568 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-21T23:46:37.602 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-certifi. 2024-09-21T23:46:37.607 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-21T23:46:37.629 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-21T23:46:37.660 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-21T23:46:37.919 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pecan. 2024-09-21T23:46:37.946 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-21T23:46:37.985 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-21T23:46:37.988 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-idna. 2024-09-21T23:46:38.015 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-21T23:46:38.055 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-21T23:46:38.349 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-certifi. 2024-09-21T23:46:38.374 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-urllib3. 2024-09-21T23:46:38.376 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-21T23:46:38.401 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-21T23:46:38.415 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-21T23:46:38.441 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-21T23:46:38.743 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-idna. 2024-09-21T23:46:38.769 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-requests. 2024-09-21T23:46:38.771 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-21T23:46:38.796 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-21T23:46:38.810 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-21T23:46:38.836 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-21T23:46:39.071 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-urllib3. 2024-09-21T23:46:39.098 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-21T23:46:39.129 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-21T23:46:39.147 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-21T23:46:39.174 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-21T23:46:39.214 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-21T23:46:39.408 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-requests. 2024-09-21T23:46:39.435 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-21T23:46:39.474 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-21T23:46:39.533 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-21T23:46:39.560 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T23:46:39.608 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mgr-modules-core (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:39.778 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-21T23:46:39.805 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-21T23:46:39.844 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-21T23:46:39.997 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-21T23:46:40.024 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:40.063 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:40.122 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-21T23:46:40.138 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T23:46:40.172 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-modules-core (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:40.399 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-21T23:46:40.426 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-21T23:46:40.466 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-21T23:46:40.544 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-21T23:46:40.571 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:40.610 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:40.735 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-openssl. 2024-09-21T23:46:40.760 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-21T23:46:40.793 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-21T23:46:40.854 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-21T23:46:40.871 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-21T23:46:40.913 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-21T23:46:41.130 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mgr. 2024-09-21T23:46:41.157 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:41.197 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mgr (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:41.241 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-openssl. 2024-09-21T23:46:41.268 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-21T23:46:41.307 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-21T23:46:41.600 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mon. 2024-09-21T23:46:41.627 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:41.666 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mon (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:41.711 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr. 2024-09-21T23:46:41.738 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:41.777 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:42.576 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mon. 2024-09-21T23:46:42.604 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:42.987 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-osd. 2024-09-21T23:46:43.014 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:43.022 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mon (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:43.045 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-osd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:43.751 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-osd. 2024-09-21T23:46:43.778 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:43.818 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-osd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:44.065 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph. 2024-09-21T23:46:44.081 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../066-ceph_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:44.123 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:44.443 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-fuse. 2024-09-21T23:46:44.470 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:44.509 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:44.846 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mds. 2024-09-21T23:46:44.873 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:44.904 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph. 2024-09-21T23:46:44.904 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mds (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:44.920 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../066-ceph_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:44.979 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:45.240 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-fuse. 2024-09-21T23:46:45.267 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:45.282 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package cephadm. 2024-09-21T23:46:45.298 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:45.309 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../069-cephadm_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:45.348 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:45.668 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-21T23:46:45.669 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mds. 2024-09-21T23:46:45.695 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-21T23:46:45.696 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:45.734 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-21T23:46:45.752 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mds (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:46.062 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-21T23:46:46.079 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-21T23:46:46.120 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-21T23:46:46.172 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package cephadm. 2024-09-21T23:46:46.199 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../069-cephadm_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:46.246 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:46.591 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-21T23:46:46.618 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-21T23:46:46.657 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-21T23:46:47.010 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-21T23:46:47.027 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-21T23:46:47.077 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-21T23:46:47.142 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-babel. 2024-09-21T23:46:47.158 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-21T23:46:47.192 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-21T23:46:47.503 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-jinja2. 2024-09-21T23:46:47.519 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-21T23:46:47.553 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-21T23:46:47.873 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-21T23:46:47.901 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T23:46:47.931 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mgr-cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:48.099 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-babel. 2024-09-21T23:46:48.126 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-21T23:46:48.165 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-21T23:46:48.518 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-21T23:46:48.527 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-jinja2. 2024-09-21T23:46:48.545 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-21T23:46:48.554 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-21T23:46:48.584 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-21T23:46:48.593 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-21T23:46:48.879 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-routes. 2024-09-21T23:46:48.906 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-21T23:46:48.945 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-21T23:46:48.971 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-21T23:46:48.999 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T23:46:49.038 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:49.299 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-21T23:46:49.327 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T23:46:49.365 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mgr-dashboard (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:49.383 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-21T23:46:49.410 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-21T23:46:49.450 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-21T23:46:49.752 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-routes. 2024-09-21T23:46:49.780 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-21T23:46:49.819 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-21T23:46:50.156 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-21T23:46:50.171 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T23:46:50.214 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-dashboard (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:50.545 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-21T23:46:50.562 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-21T23:46:50.603 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-21T23:46:51.091 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-joblib. 2024-09-21T23:46:51.118 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-21T23:46:51.157 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-21T23:46:51.478 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-21T23:46:51.506 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-21T23:46:51.519 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-21T23:46:51.547 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-21T23:46:51.552 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-21T23:46:51.585 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-21T23:46:51.880 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-sklearn. 2024-09-21T23:46:51.908 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-21T23:46:51.938 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-21T23:46:52.073 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-joblib. 2024-09-21T23:46:52.100 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-21T23:46:52.139 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-21T23:46:52.518 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-21T23:46:52.546 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-21T23:46:52.584 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-21T23:46:52.672 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-21T23:46:52.700 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T23:46:52.832 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:53.038 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-sklearn. 2024-09-21T23:46:53.066 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-21T23:46:53.112 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-21T23:46:53.819 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-cachetools. 2024-09-21T23:46:53.847 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-21T23:46:53.871 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-21T23:46:53.885 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-21T23:46:53.899 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T23:46:54.023 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:54.189 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-21T23:46:54.217 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-21T23:46:54.255 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-21T23:46:54.625 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-21T23:46:54.653 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-21T23:46:54.692 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-21T23:46:55.003 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-rsa. 2024-09-21T23:46:55.020 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-21T23:46:55.061 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-21T23:46:55.063 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cachetools. 2024-09-21T23:46:55.090 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-21T23:46:55.137 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-21T23:46:55.307 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-google-auth. 2024-09-21T23:46:55.335 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-21T23:46:55.365 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-21T23:46:55.507 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-21T23:46:55.534 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-21T23:46:55.582 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-21T23:46:55.660 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-21T23:46:55.688 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-21T23:46:55.726 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-21T23:46:55.935 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-21T23:46:55.963 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-21T23:46:56.010 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-21T23:46:56.037 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-websocket. 2024-09-21T23:46:56.065 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-21T23:46:56.104 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-21T23:46:56.322 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rsa. 2024-09-21T23:46:56.350 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-21T23:46:56.388 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-21T23:46:56.424 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-21T23:46:56.452 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-21T23:46:56.491 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-21T23:46:56.726 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-google-auth. 2024-09-21T23:46:56.754 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-21T23:46:56.792 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-21T23:46:57.104 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-21T23:46:57.132 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-21T23:46:57.170 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-21T23:46:57.171 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-21T23:46:57.188 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T23:46:57.229 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:57.498 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-websocket. 2024-09-21T23:46:57.526 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-21T23:46:57.532 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-21T23:46:57.560 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-21T23:46:57.565 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-21T23:46:57.599 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-21T23:46:57.877 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-21T23:46:57.905 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-21T23:46:57.935 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-21T23:46:57.943 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-21T23:46:57.963 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-21T23:46:58.002 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-21T23:46:58.271 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package jq. 2024-09-21T23:46:58.299 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-21T23:46:58.337 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-21T23:46:58.591 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package socat. 2024-09-21T23:46:58.591 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-21T23:46:58.608 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T23:46:58.608 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-21T23:46:58.640 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:58.799 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-21T23:46:58.935 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-21T23:46:58.963 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-21T23:46:59.001 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-21T23:46:59.085 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package xmlstarlet. 2024-09-21T23:46:59.102 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-21T23:46:59.135 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-21T23:46:59.347 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-21T23:46:59.375 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-21T23:46:59.413 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-21T23:46:59.507 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-test. 2024-09-21T23:46:59.534 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../097-ceph-test_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:46:59.572 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-test (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:46:59.657 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package jq. 2024-09-21T23:46:59.674 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-21T23:46:59.715 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-21T23:46:59.918 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package socat. 2024-09-21T23:46:59.946 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-21T23:46:59.985 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-21T23:47:00.313 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package xmlstarlet. 2024-09-21T23:47:00.342 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-21T23:47:00.380 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-21T23:47:00.734 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-test. 2024-09-21T23:47:00.750 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../097-ceph-test_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:47:00.792 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-test (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:01.927 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package ceph-volume. 2024-09-21T23:47:01.955 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T23:47:01.993 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking ceph-volume (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:02.306 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-21T23:47:02.334 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:47:02.372 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking libcephfs-dev (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:02.716 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-21T23:47:02.744 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-21T23:47:02.791 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-21T23:47:03.113 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-21T23:47:03.141 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-21T23:47:03.155 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-volume. 2024-09-21T23:47:03.179 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-21T23:47:03.183 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T23:47:03.221 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-volume (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:03.709 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package nvme-cli. 2024-09-21T23:47:03.737 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-21T23:47:03.775 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-21T23:47:03.851 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-21T23:47:03.868 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:47:03.909 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libcephfs-dev (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:04.172 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package pkg-config. 2024-09-21T23:47:04.200 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-21T23:47:04.230 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-21T23:47:04.237 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-21T23:47:04.265 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-21T23:47:04.303 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-21T23:47:04.458 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-21T23:47:04.486 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-21T23:47:04.524 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-21T23:47:04.667 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-21T23:47:04.695 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-21T23:47:04.733 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-21T23:47:04.894 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-21T23:47:04.922 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-21T23:47:04.952 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-21T23:47:05.121 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package nvme-cli. 2024-09-21T23:47:05.149 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-21T23:47:05.187 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-21T23:47:05.255 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pastescript. 2024-09-21T23:47:05.283 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-21T23:47:05.330 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-21T23:47:05.617 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package pkg-config. 2024-09-21T23:47:05.645 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-21T23:47:05.675 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pluggy. 2024-09-21T23:47:05.684 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-21T23:47:05.704 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-21T23:47:05.742 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-21T23:47:05.920 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-21T23:47:05.937 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-21T23:47:05.970 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-21T23:47:06.036 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-psutil. 2024-09-21T23:47:06.065 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-21T23:47:06.103 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-21T23:47:06.373 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-21T23:47:06.401 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-21T23:47:06.431 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-21T23:47:06.456 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-py. 2024-09-21T23:47:06.485 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-21T23:47:06.531 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-21T23:47:06.718 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pastescript. 2024-09-21T23:47:06.746 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-21T23:47:06.784 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-21T23:47:06.859 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pygments. 2024-09-21T23:47:06.888 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-21T23:47:06.926 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-21T23:47:07.121 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pluggy. 2024-09-21T23:47:07.149 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-21T23:47:07.187 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-21T23:47:07.388 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-21T23:47:07.417 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-21T23:47:07.454 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-21T23:47:07.499 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-psutil. 2024-09-21T23:47:07.527 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-21T23:47:07.565 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-21T23:47:07.758 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-toml. 2024-09-21T23:47:07.786 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-21T23:47:07.815 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-21T23:47:07.910 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-py. 2024-09-21T23:47:07.938 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-21T23:47:08.018 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-21T23:47:08.119 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-pytest. 2024-09-21T23:47:08.147 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-21T23:47:08.185 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-21T23:47:08.347 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pygments. 2024-09-21T23:47:08.375 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-21T23:47:08.413 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-21T23:47:08.505 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package python3-simplejson. 2024-09-21T23:47:08.534 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-21T23:47:08.572 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-21T23:47:08.808 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-21T23:47:08.824 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-21T23:47:08.842 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-21T23:47:08.859 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-21T23:47:08.866 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-21T23:47:08.900 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-21T23:47:09.187 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-toml. 2024-09-21T23:47:09.215 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-21T23:47:09.253 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-21T23:47:09.520 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package radosgw. 2024-09-21T23:47:09.549 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../116-radosgw_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:47:09.556 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pytest. 2024-09-21T23:47:09.578 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking radosgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:09.585 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-21T23:47:09.623 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-21T23:47:09.910 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-simplejson. 2024-09-21T23:47:09.926 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-21T23:47:09.968 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-21T23:47:10.212 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-21T23:47:10.229 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-21T23:47:10.271 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-21T23:47:10.431 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package rbd-fuse. 2024-09-21T23:47:10.448 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:47:10.489 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking rbd-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:10.817 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package rbd-mirror. 2024-09-21T23:47:10.846 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../118-rbd-mirror_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:47:10.883 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking rbd-mirror (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:11.009 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package radosgw. 2024-09-21T23:47:11.037 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../116-radosgw_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:47:11.074 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking radosgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:11.356 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-21T23:47:11.631 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-21T23:47:11.757 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-21T23:47:11.936 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package rbd-fuse. 2024-09-21T23:47:11.964 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:47:12.002 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking rbd-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:12.026 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-21T23:47:12.298 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package rbd-mirror. 2024-09-21T23:47:12.326 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../118-rbd-mirror_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T23:47:12.355 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking rbd-mirror (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:12.377 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-21T23:47:12.784 INFO:teuthology.orchestra.run.smithi031.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-21T23:47:12.811 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-21T23:47:12.842 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:13.069 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-21T23:47:13.187 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-21T23:47:13.342 INFO:teuthology.orchestra.run.smithi031.stdout:Adding system user cephadm....done 2024-09-21T23:47:13.427 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-21T23:47:13.498 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-21T23:47:13.745 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-21T23:47:13.775 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-21T23:47:14.017 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-21T23:47:14.119 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-21T23:47:14.170 INFO:teuthology.orchestra.run.smithi060.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-21T23:47:14.239 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:14.353 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-21T23:47:14.596 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-21T23:47:14.724 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-21T23:47:14.764 INFO:teuthology.orchestra.run.smithi060.stdout:Adding system user cephadm....done 2024-09-21T23:47:14.896 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-21T23:47:15.034 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-21T23:47:15.155 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-21T23:47:15.398 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-21T23:47:15.411 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-21T23:47:15.524 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-21T23:47:15.662 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-21T23:47:15.784 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-21T23:47:15.922 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-21T23:47:16.060 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-21T23:47:16.186 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-21T23:47:16.198 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-21T23:47:16.466 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-21T23:47:16.480 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-21T23:47:16.733 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-21T23:47:16.823 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-21T23:47:16.985 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-ceph-argparse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:17.083 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-21T23:47:17.235 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-21T23:47:17.342 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-21T23:47:17.495 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-21T23:47:17.596 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-21T23:47:17.610 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-21T23:47:17.697 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-21T23:47:17.862 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-21T23:47:17.956 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-21T23:47:18.113 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-21T23:47:18.343 INFO:teuthology.orchestra.run.smithi031.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-21T23:47:18.347 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-ceph-argparse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:18.371 INFO:teuthology.orchestra.run.smithi031.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-21T23:47:18.434 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-21T23:47:18.552 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-21T23:47:18.615 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-21T23:47:18.750 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-21T23:47:18.876 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-21T23:47:18.911 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-21T23:47:19.037 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-21T23:47:19.229 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-21T23:47:19.297 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-21T23:47:19.564 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-21T23:47:19.603 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-21T23:47:19.891 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up zip (3.0-12build2) ... 2024-09-21T23:47:19.966 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-21T23:47:19.992 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-21T23:47:20.016 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-21T23:47:20.048 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-21T23:47:20.199 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-21T23:47:20.584 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-21T23:47:20.718 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-21T23:47:20.743 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-21T23:47:20.986 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-21T23:47:20.994 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-21T23:47:21.095 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-21T23:47:21.196 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-21T23:47:21.254 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-21T23:47:21.497 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-21T23:47:21.547 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up zip (3.0-12build2) ... 2024-09-21T23:47:21.673 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-21T23:47:21.881 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-21T23:47:22.150 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-21T23:47:22.366 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-21T23:47:22.559 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-21T23:47:22.634 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-21T23:47:22.760 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-21T23:47:22.852 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-21T23:47:22.886 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-21T23:47:23.188 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-21T23:47:23.212 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-21T23:47:23.454 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-21T23:47:23.572 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-21T23:47:23.580 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:23.832 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-21T23:47:23.906 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-21T23:47:24.217 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-21T23:47:24.233 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-21T23:47:24.543 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-21T23:47:24.894 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-21T23:47:25.146 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-21T23:47:25.272 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:25.446 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-21T23:47:25.590 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-21T23:47:25.681 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T23:47:25.824 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-21T23:47:25.933 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-21T23:47:26.092 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-21T23:47:26.218 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-21T23:47:26.344 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-21T23:47:26.637 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-21T23:47:26.888 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-21T23:47:27.130 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T23:47:27.230 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-21T23:47:27.256 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-21T23:47:27.482 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T23:47:27.566 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-21T23:47:27.633 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-21T23:47:27.909 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-21T23:47:27.951 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-21T23:47:28.035 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-21T23:47:28.161 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-21T23:47:28.209 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-21T23:47:28.463 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-21T23:47:28.477 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-21T23:47:28.722 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-21T23:47:28.762 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-21T23:47:28.982 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T23:47:29.029 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-21T23:47:29.133 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-21T23:47:29.172 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-21T23:47:29.465 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-21T23:47:29.476 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-21T23:47:29.574 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-21T23:47:29.735 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-21T23:47:29.859 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-21T23:47:29.995 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-21T23:47:30.168 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-21T23:47:30.246 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-21T23:47:30.419 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-21T23:47:30.497 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-21T23:47:30.520 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-21T23:47:30.621 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-21T23:47:30.692 INFO:teuthology.orchestra.run.smithi031.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-21T23:47:30.782 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-21T23:47:30.933 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-21T23:47:30.939 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-21T23:47:31.065 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-21T23:47:31.284 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-21T23:47:31.410 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-21T23:47:31.467 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-21T23:47:31.736 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-21T23:47:32.018 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-21T23:47:32.046 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-21T23:47:32.152 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-21T23:47:32.289 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-21T23:47:32.411 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T23:47:32.431 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-21T23:47:32.537 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-21T23:47:32.583 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-21T23:47:32.688 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-21T23:47:32.863 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-21T23:47:32.934 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-21T23:47:33.061 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-21T23:47:33.172 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-21T23:47:33.454 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-21T23:47:33.465 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-21T23:47:33.591 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-21T23:47:33.722 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-21T23:47:33.839 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-21T23:47:33.859 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-21T23:47:34.107 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T23:47:34.118 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-21T23:47:34.258 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-21T23:47:34.472 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-21T23:47:34.626 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-21T23:47:34.865 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-21T23:47:34.927 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-21T23:47:35.141 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-21T23:47:35.229 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-21T23:47:35.355 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-21T23:47:35.494 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-21T23:47:35.620 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up librados2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:35.631 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-21T23:47:35.746 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libsqlite3-mod-ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:35.872 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-21T23:47:35.907 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-21T23:47:36.183 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libcephfs2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:36.277 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-21T23:47:36.284 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libradosstriper1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:36.385 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-21T23:47:36.663 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-21T23:47:36.670 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up librbd1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:36.829 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mgr-modules-core (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:36.955 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:36.981 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-21T23:47:37.175 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T23:47:37.175 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T23:47:37.333 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-21T23:47:37.468 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librados2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:37.594 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libsqlite3-mod-ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:37.665 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up libcephfs-dev (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:37.720 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-21T23:47:37.774 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-rados (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:37.875 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up librgw2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:37.976 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-21T23:47:38.098 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libcephfs2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:38.249 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libradosstriper1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:38.375 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-21T23:47:38.660 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librbd1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:38.803 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-modules-core (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:38.912 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:39.114 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T23:47:39.114 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T23:47:39.196 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-rbd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:39.322 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up rbd-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:39.448 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-rgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:39.574 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up python3-cephfs (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:39.580 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libcephfs-dev (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:39.700 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:39.727 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rados (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:39.857 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librgw2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:39.983 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-21T23:47:40.183 INFO:teuthology.orchestra.run.smithi031.stdout:Adding group ceph....done 2024-09-21T23:47:40.527 INFO:teuthology.orchestra.run.smithi031.stdout:Adding system user ceph....done 2024-09-21T23:47:40.696 INFO:teuthology.orchestra.run.smithi031.stdout:Setting system user ceph properties....done 2024-09-21T23:47:40.713 INFO:teuthology.orchestra.run.smithi031.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-21T23:47:40.856 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-21T23:47:41.186 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-21T23:47:41.256 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rbd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:41.382 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up rbd-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:41.550 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:41.634 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-test (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:41.675 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cephfs (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:41.760 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up radosgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:41.802 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:42.202 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T23:47:42.202 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T23:47:42.312 INFO:teuthology.orchestra.run.smithi060.stdout:Adding group ceph....done 2024-09-21T23:47:42.721 INFO:teuthology.orchestra.run.smithi060.stdout:Adding system user ceph....done 2024-09-21T23:47:42.862 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-base (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:42.890 INFO:teuthology.orchestra.run.smithi060.stdout:Setting system user ceph properties....done 2024-09-21T23:47:42.902 INFO:teuthology.orchestra.run.smithi060.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-21T23:47:43.047 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-21T23:47:43.224 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-21T23:47:43.403 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-21T23:47:43.673 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mds (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:43.878 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-test (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:43.884 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T23:47:43.884 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T23:47:44.004 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up radosgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:44.325 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up rbd-mirror (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:44.493 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T23:47:44.493 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T23:47:44.520 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T23:47:44.520 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T23:47:44.982 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-base (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:44.984 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mgr (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:45.205 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T23:47:45.206 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T23:47:45.494 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-21T23:47:45.677 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-osd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:45.974 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T23:47:45.975 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T23:47:46.010 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mds (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:46.268 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T23:47:46.269 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T23:47:46.445 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mgr-k8sevents (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:46.596 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:46.737 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up rbd-mirror (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:46.755 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mon (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:46.954 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T23:47:46.954 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T23:47:46.975 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T23:47:46.975 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T23:47:47.413 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:47.432 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mgr-cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:47.575 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:47.630 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T23:47:47.630 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T23:47:47.676 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-mgr-dashboard (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:47.935 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up ceph-volume (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:48.123 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-osd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:48.305 INFO:teuthology.orchestra.run.smithi031.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-21T23:47:48.411 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T23:47:48.411 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T23:47:48.413 INFO:teuthology.orchestra.run.smithi031.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-21T23:47:48.558 INFO:teuthology.orchestra.run.smithi031.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-21T23:47:48.908 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-k8sevents (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:49.067 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:49.235 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mon (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:49.458 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T23:47:49.458 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T23:47:49.892 INFO:teuthology.orchestra.run.smithi031.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-21T23:47:49.989 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:50.155 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:50.339 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-dashboard (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:50.507 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-volume (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T23:47:50.560 INFO:teuthology.orchestra.run.smithi031.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-21T23:47:50.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install pv 2024-09-21T23:47:50.672 INFO:teuthology.orchestra.run.smithi031.stdout:Reading package lists... 2024-09-21T23:47:50.868 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-21T23:47:50.888 INFO:teuthology.orchestra.run.smithi031.stdout:Building dependency tree... 2024-09-21T23:47:50.889 INFO:teuthology.orchestra.run.smithi031.stdout:Reading state information... 2024-09-21T23:47:50.977 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-21T23:47:51.098 INFO:teuthology.orchestra.run.smithi031.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T23:47:51.098 INFO:teuthology.orchestra.run.smithi031.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T23:47:51.098 INFO:teuthology.orchestra.run.smithi031.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T23:47:51.098 INFO:teuthology.orchestra.run.smithi031.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T23:47:51.098 INFO:teuthology.orchestra.run.smithi031.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T23:47:51.099 INFO:teuthology.orchestra.run.smithi031.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T23:47:51.099 INFO:teuthology.orchestra.run.smithi031.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T23:47:51.101 INFO:teuthology.orchestra.run.smithi031.stdout:Suggested packages: 2024-09-21T23:47:51.101 INFO:teuthology.orchestra.run.smithi031.stdout: doc-base 2024-09-21T23:47:51.130 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-21T23:47:51.130 INFO:teuthology.orchestra.run.smithi031.stdout:The following NEW packages will be installed: 2024-09-21T23:47:51.131 INFO:teuthology.orchestra.run.smithi031.stdout: pv 2024-09-21T23:47:51.186 INFO:teuthology.orchestra.run.smithi031.stdout:0 upgraded, 1 newly installed, 0 to remove and 332 not upgraded. 2024-09-21T23:47:51.186 INFO:teuthology.orchestra.run.smithi031.stdout:Need to get 44.5 kB of archives. 2024-09-21T23:47:51.186 INFO:teuthology.orchestra.run.smithi031.stdout:After this operation, 139 kB of additional disk space will be used. 2024-09-21T23:47:51.186 INFO:teuthology.orchestra.run.smithi031.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 pv amd64 1.6.6-1build2 [44.5 kB] 2024-09-21T23:47:51.569 INFO:teuthology.orchestra.run.smithi031.stdout:Fetched 44.5 kB in 0s (557 kB/s) 2024-09-21T23:47:51.636 INFO:teuthology.orchestra.run.smithi031.stdout:Selecting previously unselected package pv. 2024-09-21T23:47:51.694 INFO:teuthology.orchestra.run.smithi031.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173901 files and directories currently installed.) 2024-09-21T23:47:51.700 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing to unpack .../pv_1.6.6-1build2_amd64.deb ... 2024-09-21T23:47:51.736 INFO:teuthology.orchestra.run.smithi031.stdout:Unpacking pv (1.6.6-1build2) ... 2024-09-21T23:47:51.914 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-21T23:47:52.096 INFO:teuthology.orchestra.run.smithi031.stdout:Setting up pv (1.6.6-1build2) ... 2024-09-21T23:47:52.292 INFO:teuthology.orchestra.run.smithi031.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-21T23:47:52.567 INFO:teuthology.orchestra.run.smithi060.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-21T23:47:52.570 DEBUG:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install pv 2024-09-21T23:47:52.686 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2024-09-21T23:47:52.905 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2024-09-21T23:47:52.906 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2024-09-21T23:47:53.114 INFO:teuthology.orchestra.run.smithi060.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T23:47:53.114 INFO:teuthology.orchestra.run.smithi060.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T23:47:53.114 INFO:teuthology.orchestra.run.smithi060.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T23:47:53.115 INFO:teuthology.orchestra.run.smithi060.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T23:47:53.115 INFO:teuthology.orchestra.run.smithi060.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T23:47:53.115 INFO:teuthology.orchestra.run.smithi060.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T23:47:53.115 INFO:teuthology.orchestra.run.smithi060.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T23:47:53.117 INFO:teuthology.orchestra.run.smithi060.stdout:Suggested packages: 2024-09-21T23:47:53.117 INFO:teuthology.orchestra.run.smithi060.stdout: doc-base 2024-09-21T23:47:53.146 INFO:teuthology.orchestra.run.smithi060.stdout:The following NEW packages will be installed: 2024-09-21T23:47:53.146 INFO:teuthology.orchestra.run.smithi060.stdout: pv 2024-09-21T23:47:53.201 INFO:teuthology.orchestra.run.smithi060.stdout:0 upgraded, 1 newly installed, 0 to remove and 332 not upgraded. 2024-09-21T23:47:53.201 INFO:teuthology.orchestra.run.smithi060.stdout:Need to get 44.5 kB of archives. 2024-09-21T23:47:53.201 INFO:teuthology.orchestra.run.smithi060.stdout:After this operation, 139 kB of additional disk space will be used. 2024-09-21T23:47:53.201 INFO:teuthology.orchestra.run.smithi060.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 pv amd64 1.6.6-1build2 [44.5 kB] 2024-09-21T23:47:53.582 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 44.5 kB in 0s (554 kB/s) 2024-09-21T23:47:53.667 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package pv. 2024-09-21T23:47:53.726 INFO:teuthology.orchestra.run.smithi060.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173901 files and directories currently installed.) 2024-09-21T23:47:53.731 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../pv_1.6.6-1build2_amd64.deb ... 2024-09-21T23:47:53.783 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking pv (1.6.6-1build2) ... 2024-09-21T23:47:53.893 INFO:teuthology.orchestra.run.smithi031.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-21T23:47:53.897 DEBUG:teuthology.parallel:result is None 2024-09-21T23:47:54.186 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up pv (1.6.6-1build2) ... 2024-09-21T23:47:54.313 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-21T23:47:56.274 INFO:teuthology.orchestra.run.smithi060.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-21T23:47:56.278 DEBUG:teuthology.parallel:result is None 2024-09-21T23:47:56.279 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T23:47:56.416 DEBUG:teuthology.orchestra.run.smithi031:> dpkg-query -W -f '${Version}' ceph 2024-09-21T23:47:56.436 INFO:teuthology.orchestra.run.smithi031.stdout:19.1.1-373-g4e45e6b4-1jammy 2024-09-21T23:47:56.436 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-373-g4e45e6b4-1jammy 2024-09-21T23:47:56.436 INFO:teuthology.task.install:The correct ceph version 19.1.1-373-g4e45e6b4-1jammy is installed. 2024-09-21T23:47:56.438 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T23:47:56.574 DEBUG:teuthology.orchestra.run.smithi060:> dpkg-query -W -f '${Version}' ceph 2024-09-21T23:47:56.593 INFO:teuthology.orchestra.run.smithi060.stdout:19.1.1-373-g4e45e6b4-1jammy 2024-09-21T23:47:56.594 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-373-g4e45e6b4-1jammy 2024-09-21T23:47:56.594 INFO:teuthology.task.install:The correct ceph version 19.1.1-373-g4e45e6b4-1jammy is installed. 2024-09-21T23:47:56.595 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T23:47:56.595 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:47:56.595 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T23:47:56.610 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:47:56.610 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T23:47:56.652 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T23:47:56.652 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:47:56.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T23:47:56.666 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T23:47:56.722 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:47:56.722 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T23:47:56.736 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T23:47:56.790 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T23:47:56.790 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:47:56.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T23:47:56.803 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T23:47:56.857 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:47:56.857 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T23:47:56.871 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T23:47:56.926 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T23:47:56.927 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:47:56.927 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T23:47:56.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T23:47:56.997 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:47:56.997 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T23:47:57.010 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T23:47:57.066 INFO:teuthology.run_tasks:Running task ceph... 2024-09-21T23:47:57.135 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-21T23:47:57.135 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /var/log/ceph 2024-09-21T23:47:57.138 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 777 /var/log/ceph 2024-09-21T23:47:57.151 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-21T23:47:57.151 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T23:47:57.195 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T23:47:57.208 INFO:tasks.ceph:Creating extra log directories... 2024-09-21T23:47:57.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T23:47:57.251 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T23:47:57.275 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-21T23:47:57.275 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-21T23:47:57.275 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914544', 'branch': 'squid', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/compare-mirror-images-krbd}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7914544', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}], 'name': 'teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}, 'extra_system_packages': ['pv']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 1478, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'sleep_before_teardown': 0, 'subset': '24/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'targets': {'smithi031.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDhw5lLtglvs+U6XQ4fvivqzVMXu4Kh9MFA5QdoAONywSQ1nIxXy+vROrP/azpMzYiv2d39Yuw9pTD29gN4jbrI=', 'smithi060.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNGh2LIWkdpywhXgv5VVuZObANPS13LaR+dESgkBDuxezy2skEuA6fzULv3le0MNVsP4tfY5ZsadoZw+oPrjpgs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'extra_system_packages': ['pv']}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/compare_mirror_images.sh']}, 'env': {'RBD_DEVICE_TYPE': 'krbd'}, 'timeout': '3h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-21T23:47:57.276 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-21T23:47:57.307 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-21T23:47:57.313 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T23:47:57.355 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T23:47:57.368 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:47:57.368 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T23:47:57.414 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T23:47:57.414 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2024-09-21T23:47:57.462 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T23:47:57.462 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T23:47:57.462 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 892 Links: 1 2024-09-21T23:47:57.462 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T23:47:57.462 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-21 23:42:24.500031967 +0000 2024-09-21T23:47:57.463 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-21 23:42:24.240034289 +0000 2024-09-21T23:47:57.463 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-21 23:42:24.240034289 +0000 2024-09-21T23:47:57.463 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2024-09-21T23:47:57.463 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T23:47:57.516 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-21T23:47:57.517 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-21T23:47:57.517 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000436905 s, 1.2 MB/s 2024-09-21T23:47:57.518 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T23:47:57.568 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2024-09-21T23:47:57.614 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T23:47:57.614 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T23:47:57.614 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 908 Links: 1 2024-09-21T23:47:57.614 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T23:47:57.614 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-21 23:42:24.752029716 +0000 2024-09-21T23:47:57.615 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-21 23:42:24.748029752 +0000 2024-09-21T23:47:57.615 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-21 23:42:24.748029752 +0000 2024-09-21T23:47:57.615 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2024-09-21T23:47:57.615 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T23:47:57.669 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-21T23:47:57.669 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-21T23:47:57.669 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000370006 s, 1.4 MB/s 2024-09-21T23:47:57.670 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T23:47:57.719 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2024-09-21T23:47:57.766 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T23:47:57.766 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T23:47:57.766 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 926 Links: 1 2024-09-21T23:47:57.766 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T23:47:57.766 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-21 23:42:25.496023071 +0000 2024-09-21T23:47:57.766 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-21 23:42:25.252025249 +0000 2024-09-21T23:47:57.766 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-21 23:42:25.252025249 +0000 2024-09-21T23:47:57.767 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2024-09-21T23:47:57.767 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T23:47:57.821 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-21T23:47:57.821 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-21T23:47:57.821 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000413457 s, 1.2 MB/s 2024-09-21T23:47:57.822 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T23:47:57.871 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2024-09-21T23:47:57.918 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T23:47:57.918 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T23:47:57.918 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 937 Links: 1 2024-09-21T23:47:57.918 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T23:47:57.918 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-21 23:42:25.964018890 +0000 2024-09-21T23:47:57.918 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-21 23:42:25.716021105 +0000 2024-09-21T23:47:57.918 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-21 23:42:25.716021105 +0000 2024-09-21T23:47:57.919 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2024-09-21T23:47:57.919 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T23:47:57.972 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-21T23:47:57.972 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-21T23:47:57.972 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000292869 s, 1.7 MB/s 2024-09-21T23:47:57.974 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T23:47:58.023 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T23:47:58.023 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-21T23:47:58.024 INFO:tasks.ceph:Generating config... 2024-09-21T23:47:58.025 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-21T23:47:58.025 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-21T23:47:58.025 INFO:tasks.ceph:[client] log to stderr = False 2024-09-21T23:47:58.025 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-21T23:47:58.025 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.025 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.025 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.025 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.025 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.025 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.026 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.026 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.026 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.026 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.026 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.026 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.026 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.026 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.026 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-21T23:47:58.026 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-21T23:47:58.026 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-21T23:47:58.026 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-21T23:47:58.026 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-21T23:47:58.027 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-21T23:47:58.027 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-21T23:47:58.027 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-21T23:47:58.027 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-09-21T23:47:58.027 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-09-21T23:47:58.027 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-21T23:47:58.027 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-21T23:47:58.027 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-21T23:47:58.027 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-21T23:47:58.027 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-21T23:47:58.027 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-21T23:47:58.027 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-21T23:47:58.028 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-21T23:47:58.028 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-21T23:47:58.028 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-21T23:47:58.028 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-21T23:47:58.028 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-21T23:47:58.028 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-21T23:47:58.028 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-21T23:47:58.100 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/cluster1.keyring 2024-09-21T23:47:58.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-21T23:47:58.203 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-21T23:47:58.258 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.31')] 2024-09-21T23:47:58.259 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.31', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-21T23:47:58.259 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:47:58.259 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T23:47:58.306 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.31 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-21T23:47:58.387 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-21T23:47:58.387 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: generated fsid 432d7863-f767-472b-8b94-b82ff01e23c4 2024-09-21T23:47:58.387 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = quincy 2024-09-21T23:47:58.387 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 0 2024-09-21T23:47:58.387 INFO:teuthology.orchestra.run.smithi031.stdout:fsid 432d7863-f767-472b-8b94-b82ff01e23c4 2024-09-21T23:47:58.387 INFO:teuthology.orchestra.run.smithi031.stdout:last_changed 2024-09-21T23:47:58.390209+0000 2024-09-21T23:47:58.387 INFO:teuthology.orchestra.run.smithi031.stdout:created 2024-09-21T23:47:58.390209+0000 2024-09-21T23:47:58.387 INFO:teuthology.orchestra.run.smithi031.stdout:min_mon_release 17 (quincy) 2024-09-21T23:47:58.387 INFO:teuthology.orchestra.run.smithi031.stdout:election_strategy: 1 2024-09-21T23:47:58.387 INFO:teuthology.orchestra.run.smithi031.stdout:0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-09-21T23:47:58.387 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-21T23:47:58.388 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T23:47:58.434 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 432d7863-f767-472b-8b94-b82ff01e23c4... 2024-09-21T23:47:58.435 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-21T23:47:58.479 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-21T23:47:58.507 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2024-09-21T23:47:58.507 INFO:teuthology.orchestra.run.smithi031.stdout: chdir = "" 2024-09-21T23:47:58.507 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T23:47:58.508 INFO:teuthology.orchestra.run.smithi031.stdout: auth supported = cephx 2024-09-21T23:47:58.508 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.508 INFO:teuthology.orchestra.run.smithi031.stdout: filestore xattr use omap = true 2024-09-21T23:47:58.508 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.508 INFO:teuthology.orchestra.run.smithi031.stdout: mon clock drift allowed = 1.000 2024-09-21T23:47:58.508 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.508 INFO:teuthology.orchestra.run.smithi031.stdout: osd crush chooseleaf type = 0 2024-09-21T23:47:58.508 INFO:teuthology.orchestra.run.smithi031.stdout: auth debug = true 2024-09-21T23:47:58.508 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.508 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on old message = true 2024-09-21T23:47:58.508 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on bug = true 2024-09-21T23:47:58.508 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.508 INFO:teuthology.orchestra.run.smithi031.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T23:47:58.508 INFO:teuthology.orchestra.run.smithi031.stdout: mon pg warn max object skew = 0 2024-09-21T23:47:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T23:47:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T23:47:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default size = 2 2024-09-21T23:47:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow primary affinity = true 2024-09-21T23:47:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow pg remap = true 2024-09-21T23:47:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on legacy crush tunables = false 2024-09-21T23:47:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on crush straw calc version zero = false 2024-09-21T23:47:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on no sortbitwise = false 2024-09-21T23:47:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on osd down out interval zero = false 2024-09-21T23:47:58.509 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on too few osds = false 2024-09-21T23:47:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T23:47:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T23:47:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: mon_allow_pool_size_one = true 2024-09-21T23:47:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T23:47:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: osd default data pool replay window = 5 2024-09-21T23:47:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: mon allow pool delete = true 2024-09-21T23:47:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: mon cluster log file level = debug 2024-09-21T23:47:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: debug asserts on shutdown = true 2024-09-21T23:47:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: mon health detail to clog = false 2024-09-21T23:47:58.510 INFO:teuthology.orchestra.run.smithi031.stdout: mon host = 172.21.15.31 2024-09-21T23:47:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: mon client directed command retry = 5 2024-09-21T23:47:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on pool no app = False 2024-09-21T23:47:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject socket failures = 5000 2024-09-21T23:47:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = 432d7863-f767-472b-8b94-b82ff01e23c4 2024-09-21T23:47:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.511 INFO:teuthology.orchestra.run.smithi031.stdout:[osd] 2024-09-21T23:47:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: osd journal size = 100 2024-09-21T23:47:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub load threshold = 5.0 2024-09-21T23:47:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub max interval = 600 2024-09-21T23:47:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock profile = high_recovery_ops 2024-09-21T23:47:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: osd recover clone overlap = true 2024-09-21T23:47:58.511 INFO:teuthology.orchestra.run.smithi031.stdout: osd recovery max chunk = 1048576 2024-09-21T23:47:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug shutdown = true 2024-09-21T23:47:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug op order = true 2024-09-21T23:47:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify stray on activate = true 2024-09-21T23:47:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug trim objects = true 2024-09-21T23:47:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: osd open classes on start = true 2024-09-21T23:47:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug pg log writeout = true 2024-09-21T23:47:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: osd deep scrub update digest min age = 30 2024-09-21T23:47:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: osd map max advance = 10 2024-09-21T23:47:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.512 INFO:teuthology.orchestra.run.smithi031.stdout: journal zero on create = true 2024-09-21T23:47:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: filestore ondisk finisher threads = 3 2024-09-21T23:47:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: filestore apply finisher threads = 3 2024-09-21T23:47:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: bdev debug aio = true 2024-09-21T23:47:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug misdirected ops = true 2024-09-21T23:47:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: bluefs allocator = hybrid 2024-09-21T23:47:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore allocator = hybrid 2024-09-21T23:47:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore block size = 96636764160 2024-09-21T23:47:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore fsck on mount = True 2024-09-21T23:47:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluefs = 1/20 2024-09-21T23:47:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluestore = 1/20 2024-09-21T23:47:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-09-21T23:47:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 20 2024-09-21T23:47:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: debug rocksdb = 4/10 2024-09-21T23:47:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T23:47:58.514 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd full ratio = 0.9 2024-09-21T23:47:58.514 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd nearfull ratio = 0.8 2024-09-21T23:47:58.514 INFO:teuthology.orchestra.run.smithi031.stdout: osd failsafe full ratio = 0.95 2024-09-21T23:47:58.514 INFO:teuthology.orchestra.run.smithi031.stdout: osd objectstore = bluestore 2024-09-21T23:47:58.514 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.514 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr] 2024-09-21T23:47:58.514 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-09-21T23:47:58.514 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 20 2024-09-21T23:47:58.514 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2024-09-21T23:47:58.514 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2024-09-21T23:47:58.514 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2024-09-21T23:47:58.514 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2024-09-21T23:47:58.514 INFO:teuthology.orchestra.run.smithi031.stdout: mgr/telemetry/nag = false 2024-09-21T23:47:58.514 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.514 INFO:teuthology.orchestra.run.smithi031.stdout:[mon] 2024-09-21T23:47:58.515 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-09-21T23:47:58.515 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2024-09-21T23:47:58.515 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 20 2024-09-21T23:47:58.515 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2024-09-21T23:47:58.515 INFO:teuthology.orchestra.run.smithi031.stdout: mon data avail warn = 5 2024-09-21T23:47:58.515 INFO:teuthology.orchestra.run.smithi031.stdout: mon mgr mkfs grace = 240 2024-09-21T23:47:58.515 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2024-09-21T23:47:58.515 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd reporter subtree level = osd 2024-09-21T23:47:58.515 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd prime pg temp = true 2024-09-21T23:47:58.515 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2024-09-21T23:47:58.515 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.515 INFO:teuthology.orchestra.run.smithi031.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T23:47:58.515 INFO:teuthology.orchestra.run.smithi031.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T23:47:58.515 INFO:teuthology.orchestra.run.smithi031.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T23:47:58.515 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.516 INFO:teuthology.orchestra.run.smithi031.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T23:47:58.516 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T23:47:58.516 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T23:47:58.516 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.516 INFO:teuthology.orchestra.run.smithi031.stdout: # 1m isn't quite enough 2024-09-21T23:47:58.516 INFO:teuthology.orchestra.run.smithi031.stdout: mon_down_mkfs_grace = 2m 2024-09-21T23:47:58.516 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.516 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_filestore_osds = false 2024-09-21T23:47:58.516 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:47:58.516 INFO:teuthology.orchestra.run.smithi031.stdout:[client] 2024-09-21T23:47:58.516 INFO:teuthology.orchestra.run.smithi031.stdout: rgw cache enabled = true 2024-09-21T23:47:58.516 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable ops log = true 2024-09-21T23:47:58.516 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable usage log = true 2024-09-21T23:47:58.516 INFO:teuthology.orchestra.run.smithi031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T23:47:58.516 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T23:47:58.516 INFO:teuthology.orchestra.run.smithi031.stdout: debug rbd = 20 2024-09-21T23:47:58.517 INFO:teuthology.orchestra.run.smithi031.stdout: debug rbd_mirror = 15 2024-09-21T23:47:58.517 INFO:teuthology.orchestra.run.smithi031.stdout: log to stderr = False 2024-09-21T23:47:58.517 INFO:teuthology.orchestra.run.smithi031.stdout: rbd default features = 125 2024-09-21T23:47:58.517 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.a] 2024-09-21T23:47:58.517 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.0] 2024-09-21T23:47:58.517 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.517 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.517 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.1] 2024-09-21T23:47:58.517 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.517 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.517 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.2] 2024-09-21T23:47:58.517 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.517 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.517 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.3] 2024-09-21T23:47:58.517 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.517 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.517 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.4] 2024-09-21T23:47:58.518 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.518 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.518 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.5] 2024-09-21T23:47:58.518 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.518 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.518 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.6] 2024-09-21T23:47:58.518 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.518 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.518 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2024-09-21T23:47:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2024-09-21T23:47:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T23:47:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2024-09-21T23:47:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2024-09-21T23:47:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2024-09-21T23:47:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2024-09-21T23:47:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2024-09-21T23:47:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2024-09-21T23:47:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2024-09-21T23:47:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T23:47:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2024-09-21T23:47:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T23:47:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T23:47:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2024-09-21T23:47:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2024-09-21T23:47:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2024-09-21T23:47:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2024-09-21T23:47:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2024-09-21T23:47:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2024-09-21T23:47:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2024-09-21T23:47:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2024-09-21T23:47:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T23:47:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T23:47:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: mon_allow_pool_size_one = true 2024-09-21T23:47:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T23:47:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2024-09-21T23:47:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2024-09-21T23:47:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2024-09-21T23:47:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2024-09-21T23:47:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: mon health detail to clog = false 2024-09-21T23:47:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = 172.21.15.31 2024-09-21T23:47:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: mon client directed command retry = 5 2024-09-21T23:47:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on pool no app = False 2024-09-21T23:47:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: ms inject socket failures = 5000 2024-09-21T23:47:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = 432d7863-f767-472b-8b94-b82ff01e23c4 2024-09-21T23:47:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.522 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2024-09-21T23:47:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2024-09-21T23:47:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2024-09-21T23:47:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2024-09-21T23:47:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: osd mclock profile = high_recovery_ops 2024-09-21T23:47:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2024-09-21T23:47:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2024-09-21T23:47:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2024-09-21T23:47:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2024-09-21T23:47:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2024-09-21T23:47:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug trim objects = true 2024-09-21T23:47:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2024-09-21T23:47:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2024-09-21T23:47:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2024-09-21T23:47:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2024-09-21T23:47:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2024-09-21T23:47:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2024-09-21T23:47:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2024-09-21T23:47:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2024-09-21T23:47:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2024-09-21T23:47:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: bluefs allocator = hybrid 2024-09-21T23:47:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore allocator = hybrid 2024-09-21T23:47:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore block size = 96636764160 2024-09-21T23:47:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore fsck on mount = True 2024-09-21T23:47:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluefs = 1/20 2024-09-21T23:47:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluestore = 1/20 2024-09-21T23:47:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-09-21T23:47:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 20 2024-09-21T23:47:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: debug rocksdb = 4/10 2024-09-21T23:47:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T23:47:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd full ratio = 0.9 2024-09-21T23:47:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd nearfull ratio = 0.8 2024-09-21T23:47:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: osd failsafe full ratio = 0.95 2024-09-21T23:47:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: osd objectstore = bluestore 2024-09-21T23:47:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.525 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2024-09-21T23:47:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-09-21T23:47:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2024-09-21T23:47:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-09-21T23:47:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-09-21T23:47:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-09-21T23:47:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-09-21T23:47:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2024-09-21T23:47:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.525 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2024-09-21T23:47:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-09-21T23:47:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-09-21T23:47:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2024-09-21T23:47:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-09-21T23:47:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2024-09-21T23:47:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2024-09-21T23:47:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-09-21T23:47:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2024-09-21T23:47:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2024-09-21T23:47:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-09-21T23:47:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T23:47:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T23:47:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T23:47:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T23:47:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T23:47:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T23:47:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: # 1m isn't quite enough 2024-09-21T23:47:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: mon_down_mkfs_grace = 2m 2024-09-21T23:47:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_filestore_osds = false 2024-09-21T23:47:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:47:58.527 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2024-09-21T23:47:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2024-09-21T23:47:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2024-09-21T23:47:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2024-09-21T23:47:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T23:47:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T23:47:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: debug rbd = 20 2024-09-21T23:47:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: debug rbd_mirror = 15 2024-09-21T23:47:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: log to stderr = False 2024-09-21T23:47:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: rbd default features = 125 2024-09-21T23:47:58.527 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2024-09-21T23:47:58.527 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.0] 2024-09-21T23:47:58.528 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.528 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.528 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.1] 2024-09-21T23:47:58.528 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.528 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.528 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.2] 2024-09-21T23:47:58.528 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.528 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.528 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.3] 2024-09-21T23:47:58.528 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.528 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.528 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.4] 2024-09-21T23:47:58.528 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.528 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.528 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.5] 2024-09-21T23:47:58.529 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.529 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.529 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.6] 2024-09-21T23:47:58.529 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:47:58.529 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:47:58.530 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-21T23:47:58.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-21T23:47:58.623 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-21T23:47:58.623 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:47:58.623 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-21T23:47:58.628 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:47:58.629 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-21T23:47:58.674 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi031.front.sepia.ceph.com 2024-09-21T23:47:58.674 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:47:58.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-21T23:47:58.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-21T23:47:58.739 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:47:58.740 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-21T23:47:58.786 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:47:58.786 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:47:58.786 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-21T23:47:58.786 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-21T23:47:58.812 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:47:58.812 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-21T23:47:58.859 INFO:tasks.ceph:Setting up mon nodes... 2024-09-21T23:47:58.859 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-21T23:47:58.859 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-21T23:47:58.905 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-21T23:47:58.910 INFO:tasks.ceph:Setting up mds nodes... 2024-09-21T23:47:58.910 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-21T23:47:58.910 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-21T23:47:58.986 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-21T23:47:58.998 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-21T23:47:59.036 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-21T23:47:59.060 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-21T23:47:59.136 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-21T23:47:59.148 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-21T23:47:59.227 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-21T23:47:59.239 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-21T23:47:59.314 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-21T23:47:59.324 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-21T23:47:59.398 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-21T23:47:59.410 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-21T23:47:59.489 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-21T23:47:59.500 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-21T23:47:59.575 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-21T23:47:59.587 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-21T23:47:59.662 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-21T23:47:59.672 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-21T23:47:59.672 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-21T23:47:59.672 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-21T23:47:59.687 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T23:47:59.687 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-21T23:47:59.687 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-21T23:47:59.687 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T23:47:59.750 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T23:47:59.750 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T23:47:59.750 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T23:47:59.750 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T23:47:59.750 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T23:47:59.750 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-21T23:47:59.750 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T23:47:59.750 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T23:47:59.751 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T23:47:59.751 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T23:48:00.201 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-21T23:48:00.203 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-21T23:48:00.203 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-21T23:48:00.297 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-21T23:48:00.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T23:48:00.306 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T23:48:00.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-21T23:48:00.359 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T23:48:00.359 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-21T23:48:00.359 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-21T23:48:00.359 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T23:48:00.421 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T23:48:00.421 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T23:48:00.421 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T23:48:00.421 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T23:48:00.421 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T23:48:00.421 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-21T23:48:00.421 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T23:48:00.422 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T23:48:00.422 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T23:48:00.422 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T23:48:00.881 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-21T23:48:00.883 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-21T23:48:00.884 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-21T23:48:01.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-21T23:48:01.046 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T23:48:01.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T23:48:01.047 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-21T23:48:01.102 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T23:48:01.102 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-21T23:48:01.103 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-21T23:48:01.103 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T23:48:01.165 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T23:48:01.165 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T23:48:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T23:48:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T23:48:01.166 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T23:48:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-21T23:48:01.166 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T23:48:01.166 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T23:48:01.166 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T23:48:01.166 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T23:48:01.620 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-21T23:48:01.623 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-21T23:48:01.623 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-21T23:48:01.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-21T23:48:01.786 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T23:48:01.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T23:48:01.787 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-21T23:48:01.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:48:01.899+0000 7efd5ea3d7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-21T23:48:01.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:48:01.899+0000 7efd5ea3d7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-21T23:48:01.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:48:01.899+0000 7efd5ea3d7c0 -1 bdev(0x555bcaa25800 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2024-09-21T23:48:01.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:48:01.899+0000 7efd5ea3d7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-09-21T23:48:05.061 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-21T23:48:05.075 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-21T23:48:05.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:48:05.158+0000 7f89ca2047c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-21T23:48:05.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:48:05.158+0000 7f89ca2047c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-21T23:48:05.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:48:05.158+0000 7f89ca2047c0 -1 bdev(0x5572cc9a5800 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2024-09-21T23:48:05.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:48:05.158+0000 7f89ca2047c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-09-21T23:48:08.298 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-21T23:48:08.354 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-21T23:48:08.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:48:08.438+0000 7f4770a687c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-21T23:48:08.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:48:08.438+0000 7f4770a687c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-21T23:48:08.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:48:08.438+0000 7f4770a687c0 -1 bdev(0x555d9266d800 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2024-09-21T23:48:08.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-21T23:48:08.438+0000 7f4770a687c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-09-21T23:48:11.597 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-21T23:48:11.654 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-21T23:48:11.655 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:48:11.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-21T23:48:11.710 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:48:11.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-21T23:48:11.765 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:48:11.765 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-21T23:48:11.821 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:48:11.821 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-21T23:48:11.877 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:48:11.877 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-21T23:48:11.926 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:11.926 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-21T23:48:11.932 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:11.933 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-21T23:48:11.979 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:11.979 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-21T23:48:12.027 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:12.027 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-21T23:48:12.075 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:12.075 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-21T23:48:12.123 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:12.123 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-21T23:48:12.171 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:12.171 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-21T23:48:12.219 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:12.219 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-21T23:48:12.267 INFO:tasks.ceph:Adding keys to all mons... 2024-09-21T23:48:12.267 DEBUG:teuthology.orchestra.run.smithi031:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-21T23:48:12.279 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.x] 2024-09-21T23:48:12.279 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQAuW+9mWLIqNhAARMxqT6wYaOXVP6965RAebQ== 2024-09-21T23:48:12.279 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.0] 2024-09-21T23:48:12.279 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQAxW+9mgb7HNRAA14Lc6IaDjY3CjMnO+TaB1g== 2024-09-21T23:48:12.279 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.1] 2024-09-21T23:48:12.279 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQA1W+9m2vSlCRAAUJ8JwSH3x07ksE/riibW9g== 2024-09-21T23:48:12.280 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.2] 2024-09-21T23:48:12.280 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQA4W+9myWFZGhAAm7jX4xYsnjC1kgDuW8Txbg== 2024-09-21T23:48:12.280 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-09-21T23:48:12.280 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQAuW+9m4OgJOxAAujOo5Uka4QoKk9YPtkkpoA== 2024-09-21T23:48:12.280 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror] 2024-09-21T23:48:12.280 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQAvW+9mVwhRAhAAv3lMBhBssBkuQcr7V3TJaA== 2024-09-21T23:48:12.280 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.0] 2024-09-21T23:48:12.280 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQAvW+9m2h1RCBAAMAmYHS0D8IXT1ojXA+aT1g== 2024-09-21T23:48:12.280 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.1] 2024-09-21T23:48:12.280 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQAvW+9mOxC4DRAAYgGf6X6pKLUp9W16J8vU5w== 2024-09-21T23:48:12.280 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.2] 2024-09-21T23:48:12.280 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQAvW+9mZ1PuEhAAZ3GKlWlz79R0Xg/RzDafWA== 2024-09-21T23:48:12.280 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.3] 2024-09-21T23:48:12.280 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQAvW+9msPDxFxAARXF5nbGlg0aQKrR9i58lOw== 2024-09-21T23:48:12.280 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.4] 2024-09-21T23:48:12.280 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQAvW+9m5tRVHRAA1vmM21fxPjNZffmsHIplag== 2024-09-21T23:48:12.281 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.5] 2024-09-21T23:48:12.281 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQAvW+9mDcV7IhAACVd+QdvIxr23YBcsJEhV5A== 2024-09-21T23:48:12.281 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.6] 2024-09-21T23:48:12.281 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQAvW+9miQOrJxAAJQqmRh/nc09M9zj5phiZ6A== 2024-09-21T23:48:12.282 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T23:48:12.372 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:48:12.415 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:48:12.499 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:48:12.623 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:48:12.671 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:48:12.756 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:48:12.845 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:48:12.888 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:48:12.975 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:48:13.020 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:48:13.105 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:48:13.197 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:48:13.245 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-21T23:48:13.245 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-21T23:48:13.256 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-21T23:48:13.344 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-21T23:48:13.355 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-21T23:48:13.406 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-21T23:48:13.406 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-21T23:48:13.406 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-21T23:48:13.451 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-21T23:48:13.451 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-21T23:48:13.451 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-21T23:48:13.451 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-21T23:48:13.453 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-21T23:48:13.453 INFO:tasks.ceph:Setting crush tunables to default 2024-09-21T23:48:13.453 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-21T23:48:13.695 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2024-09-21T23:48:13.714 INFO:tasks.ceph:check_enable_crimson: False 2024-09-21T23:48:13.714 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-21T23:48:13.714 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:48:13.714 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-21T23:48:13.725 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:48:13.725 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-21T23:48:13.785 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:48:13.785 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-21T23:48:13.840 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:13.838+0000 7f79d0064180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T23:48:13.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster cluster1 osd new 517da62b-efb1-42ab-ae53-78dde46826da 0 2024-09-21T23:48:14.099 INFO:teuthology.orchestra.run.smithi031.stdout:0 2024-09-21T23:48:14.117 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster cluster1 osd new 4aa1c944-f2cb-4995-ba36-159cd9e3c1aa 1 2024-09-21T23:48:14.325 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-09-21T23:48:14.344 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster cluster1 osd new 9eee3c58-3c7a-43f6-9684-6fecbcc505ff 2 2024-09-21T23:48:14.373 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:14.370+0000 7f79d0064180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T23:48:14.546 INFO:teuthology.orchestra.run.smithi031.stdout:2 2024-09-21T23:48:14.565 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-21T23:48:14.565 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-21T23:48:14.607 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-21T23:48:14.607 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-21T23:48:14.607 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-21T23:48:14.608 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:14.606+0000 7f79d0064180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T23:48:14.609 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-21T23:48:14.609 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-21T23:48:14.609 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-21T23:48:14.611 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-21T23:48:14.611 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-21T23:48:14.802 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:14.802 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":5,"fsid":"432d7863-f767-472b-8b94-b82ff01e23c4","created":"2024-09-21T23:48:13.596459+0000","modified":"2024-09-21T23:48:14.548448+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T23:48:14.548448+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"517da62b-efb1-42ab-ae53-78dde46826da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4aa1c944-f2cb-4995-ba36-159cd9e3c1aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9eee3c58-3c7a-43f6-9684-6fecbcc505ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:48:14.818 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-21T23:48:14.818 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-21T23:48:15.120 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-21T23:48:15.226 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:15.226+0000 7f79d0064180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T23:48:15.313 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:15.314+0000 7f79d0064180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T23:48:15.367 INFO:teuthology.misc.health.smithi031.stdout: 2024-09-21T23:48:15.367 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":5,"fsid":"432d7863-f767-472b-8b94-b82ff01e23c4","created":"2024-09-21T23:48:13.596459+0000","modified":"2024-09-21T23:48:14.548448+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T23:48:14.548448+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"517da62b-efb1-42ab-ae53-78dde46826da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4aa1c944-f2cb-4995-ba36-159cd9e3c1aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9eee3c58-3c7a-43f6-9684-6fecbcc505ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:48:15.388 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-21T23:48:15.399 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:15.398+0000 7f79d0064180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T23:48:15.480 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:15.478+0000 7f79d0064180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T23:48:15.535 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:15.534+0000 7f79d0064180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T23:48:15.616 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:15.614+0000 7f79d0064180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T23:48:15.671 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:15.670+0000 7f79d0064180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T23:48:15.829 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:15.826+0000 7f79d0064180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T23:48:15.894 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:15.894+0000 7f79d0064180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T23:48:16.003 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-21T23:48:16.002+0000 7f1bdc61d7c0 -1 Falling back to public interface 2024-09-21T23:48:16.031 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:48:16.030+0000 7f55033a67c0 -1 Falling back to public interface 2024-09-21T23:48:16.163 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-21T23:48:16.162+0000 7fab7042a7c0 -1 Falling back to public interface 2024-09-21T23:48:16.559 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-21T23:48:16.559 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-21T23:48:16.559 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr: from numpy import show_config as show_numpy_config 2024-09-21T23:48:16.629 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:16.626+0000 7f79d0064180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T23:48:16.786 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:16.786+0000 7f79d0064180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T23:48:16.962 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:16.962+0000 7f79d0064180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T23:48:17.028 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:17.026+0000 7f79d0064180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T23:48:17.171 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:17.170+0000 7f79d0064180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T23:48:17.800 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:17.798+0000 7f79d0064180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T23:48:17.879 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:17.878+0000 7f79d0064180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T23:48:18.071 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:18.070+0000 7f79d0064180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T23:48:18.148 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:18.146+0000 7f79d0064180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T23:48:18.454 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:18.454+0000 7f79d0064180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T23:48:18.519 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:18.518+0000 7f79d0064180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T23:48:19.549 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2024-09-21T23:48:19.550+0000 7f7976b98640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-21T23:48:20.088 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:48:20.086+0000 7f55033a67c0 -1 osd.2 0 log_to_monitors true 2024-09-21T23:48:20.124 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-21T23:48:20.122+0000 7f1bdc61d7c0 -1 osd.1 0 log_to_monitors true 2024-09-21T23:48:20.248 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-21T23:48:20.246+0000 7fab7042a7c0 -1 osd.0 0 log_to_monitors true 2024-09-21T23:48:21.691 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-21T23:48:22.031 INFO:teuthology.misc.health.smithi031.stdout: 2024-09-21T23:48:22.031 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":7,"fsid":"432d7863-f767-472b-8b94-b82ff01e23c4","created":"2024-09-21T23:48:13.596459+0000","modified":"2024-09-21T23:48:21.605258+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T23:48:14.548448+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"517da62b-efb1-42ab-ae53-78dde46826da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4aa1c944-f2cb-4995-ba36-159cd9e3c1aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9eee3c58-3c7a-43f6-9684-6fecbcc505ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:48:22.049 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-21T23:48:23.448 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:48:23.446+0000 7f54ffb61640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T23:48:23.455 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-21T23:48:23.454+0000 7fab6cbe5640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T23:48:23.474 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:48:23.474+0000 7f54f7184640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T23:48:23.486 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-21T23:48:23.486+0000 7fab64208640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T23:48:23.647 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-21T23:48:23.646+0000 7f1bd8dd8640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T23:48:23.678 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-21T23:48:23.678+0000 7f1bd03fb640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T23:48:28.351 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-21T23:48:28.668 INFO:teuthology.misc.health.smithi031.stdout: 2024-09-21T23:48:28.668 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":11,"fsid":"432d7863-f767-472b-8b94-b82ff01e23c4","created":"2024-09-21T23:48:13.596459+0000","modified":"2024-09-21T23:48:27.465496+0000","last_up_change":"2024-09-21T23:48:24.461211+0000","last_in_change":"2024-09-21T23:48:14.548448+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T23:48:24.627658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"517da62b-efb1-42ab-ae53-78dde46826da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":478776314},{"type":"v1","addr":"172.21.15.31:6817","nonce":478776314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":478776314},{"type":"v1","addr":"172.21.15.31:6819","nonce":478776314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":478776314},{"type":"v1","addr":"172.21.15.31:6823","nonce":478776314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":478776314},{"type":"v1","addr":"172.21.15.31:6821","nonce":478776314}]},"public_addr":"172.21.15.31:6817/478776314","cluster_addr":"172.21.15.31:6819/478776314","heartbeat_back_addr":"172.21.15.31:6823/478776314","heartbeat_front_addr":"172.21.15.31:6821/478776314","state":["exists","up"]},{"osd":1,"uuid":"4aa1c944-f2cb-4995-ba36-159cd9e3c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":437696049},{"type":"v1","addr":"172.21.15.31:6801","nonce":437696049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":437696049},{"type":"v1","addr":"172.21.15.31:6803","nonce":437696049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":437696049},{"type":"v1","addr":"172.21.15.31:6807","nonce":437696049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":437696049},{"type":"v1","addr":"172.21.15.31:6805","nonce":437696049}]},"public_addr":"172.21.15.31:6801/437696049","cluster_addr":"172.21.15.31:6803/437696049","heartbeat_back_addr":"172.21.15.31:6807/437696049","heartbeat_front_addr":"172.21.15.31:6805/437696049","state":["exists","up"]},{"osd":2,"uuid":"9eee3c58-3c7a-43f6-9684-6fecbcc505ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1610229797},{"type":"v1","addr":"172.21.15.31:6809","nonce":1610229797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1610229797},{"type":"v1","addr":"172.21.15.31:6811","nonce":1610229797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1610229797},{"type":"v1","addr":"172.21.15.31:6815","nonce":1610229797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1610229797},{"type":"v1","addr":"172.21.15.31:6813","nonce":1610229797}]},"public_addr":"172.21.15.31:6809/1610229797","cluster_addr":"172.21.15.31:6811/1610229797","heartbeat_back_addr":"172.21.15.31:6815/1610229797","heartbeat_front_addr":"172.21.15.31:6813/1610229797","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:21.292592+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:21.109724+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:21.056950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:48:28.686 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-21T23:48:28.686 INFO:tasks.ceph:Creating RBD pool 2024-09-21T23:48:28.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-21T23:48:29.476 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'rbd' created 2024-09-21T23:48:29.507 DEBUG:teuthology.orchestra.run.smithi031:> rbd --cluster cluster1 pool init rbd 2024-09-21T23:48:32.535 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-21T23:48:32.535 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-21T23:48:32.535 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-21T23:48:32.536 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-21T23:48:32.536 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-21T23:48:32.889 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:32.908 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2907326323},{"type":"v1","addr":"172.21.15.31:6825","nonce":2907326323}]},"active_addr":"172.21.15.31:6825/2907326323","active_change":"2024-09-21T23:48:18.542525+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":1150523964}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2166575156}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":1061167307}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":1482063958}]}]} 2024-09-21T23:48:32.910 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-21T23:48:32.910 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-21T23:48:32.910 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-21T23:48:33.218 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:33.218 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"fsid":"432d7863-f767-472b-8b94-b82ff01e23c4","created":"2024-09-21T23:48:13.596459+0000","modified":"2024-09-21T23:48:32.497869+0000","last_up_change":"2024-09-21T23:48:24.461211+0000","last_in_change":"2024-09-21T23:48:14.548448+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T23:48:24.627658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T23:48:29.046255+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"517da62b-efb1-42ab-ae53-78dde46826da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":478776314},{"type":"v1","addr":"172.21.15.31:6817","nonce":478776314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":478776314},{"type":"v1","addr":"172.21.15.31:6819","nonce":478776314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":478776314},{"type":"v1","addr":"172.21.15.31:6823","nonce":478776314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":478776314},{"type":"v1","addr":"172.21.15.31:6821","nonce":478776314}]},"public_addr":"172.21.15.31:6817/478776314","cluster_addr":"172.21.15.31:6819/478776314","heartbeat_back_addr":"172.21.15.31:6823/478776314","heartbeat_front_addr":"172.21.15.31:6821/478776314","state":["exists","up"]},{"osd":1,"uuid":"4aa1c944-f2cb-4995-ba36-159cd9e3c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":437696049},{"type":"v1","addr":"172.21.15.31:6801","nonce":437696049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":437696049},{"type":"v1","addr":"172.21.15.31:6803","nonce":437696049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":437696049},{"type":"v1","addr":"172.21.15.31:6807","nonce":437696049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":437696049},{"type":"v1","addr":"172.21.15.31:6805","nonce":437696049}]},"public_addr":"172.21.15.31:6801/437696049","cluster_addr":"172.21.15.31:6803/437696049","heartbeat_back_addr":"172.21.15.31:6807/437696049","heartbeat_front_addr":"172.21.15.31:6805/437696049","state":["exists","up"]},{"osd":2,"uuid":"9eee3c58-3c7a-43f6-9684-6fecbcc505ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1610229797},{"type":"v1","addr":"172.21.15.31:6809","nonce":1610229797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1610229797},{"type":"v1","addr":"172.21.15.31:6811","nonce":1610229797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1610229797},{"type":"v1","addr":"172.21.15.31:6815","nonce":1610229797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1610229797},{"type":"v1","addr":"172.21.15.31:6813","nonce":1610229797}]},"public_addr":"172.21.15.31:6809/1610229797","cluster_addr":"172.21.15.31:6811/1610229797","heartbeat_back_addr":"172.21.15.31:6815/1610229797","heartbeat_front_addr":"172.21.15.31:6813/1610229797","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:21.292592+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:21.109724+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:21.056950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:48:33.237 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-21T23:48:33.237 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-21T23:48:33.550 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:33.550 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"fsid":"432d7863-f767-472b-8b94-b82ff01e23c4","created":"2024-09-21T23:48:13.596459+0000","modified":"2024-09-21T23:48:32.497869+0000","last_up_change":"2024-09-21T23:48:24.461211+0000","last_in_change":"2024-09-21T23:48:14.548448+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T23:48:24.627658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T23:48:29.046255+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"517da62b-efb1-42ab-ae53-78dde46826da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":478776314},{"type":"v1","addr":"172.21.15.31:6817","nonce":478776314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":478776314},{"type":"v1","addr":"172.21.15.31:6819","nonce":478776314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":478776314},{"type":"v1","addr":"172.21.15.31:6823","nonce":478776314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":478776314},{"type":"v1","addr":"172.21.15.31:6821","nonce":478776314}]},"public_addr":"172.21.15.31:6817/478776314","cluster_addr":"172.21.15.31:6819/478776314","heartbeat_back_addr":"172.21.15.31:6823/478776314","heartbeat_front_addr":"172.21.15.31:6821/478776314","state":["exists","up"]},{"osd":1,"uuid":"4aa1c944-f2cb-4995-ba36-159cd9e3c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":437696049},{"type":"v1","addr":"172.21.15.31:6801","nonce":437696049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":437696049},{"type":"v1","addr":"172.21.15.31:6803","nonce":437696049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":437696049},{"type":"v1","addr":"172.21.15.31:6807","nonce":437696049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":437696049},{"type":"v1","addr":"172.21.15.31:6805","nonce":437696049}]},"public_addr":"172.21.15.31:6801/437696049","cluster_addr":"172.21.15.31:6803/437696049","heartbeat_back_addr":"172.21.15.31:6807/437696049","heartbeat_front_addr":"172.21.15.31:6805/437696049","state":["exists","up"]},{"osd":2,"uuid":"9eee3c58-3c7a-43f6-9684-6fecbcc505ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1610229797},{"type":"v1","addr":"172.21.15.31:6809","nonce":1610229797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1610229797},{"type":"v1","addr":"172.21.15.31:6811","nonce":1610229797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1610229797},{"type":"v1","addr":"172.21.15.31:6815","nonce":1610229797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1610229797},{"type":"v1","addr":"172.21.15.31:6813","nonce":1610229797}]},"public_addr":"172.21.15.31:6809/1610229797","cluster_addr":"172.21.15.31:6811/1610229797","heartbeat_back_addr":"172.21.15.31:6815/1610229797","heartbeat_front_addr":"172.21.15.31:6813/1610229797","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:21.292592+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:21.109724+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:21.056950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:48:33.569 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-21T23:48:33.570 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-21T23:48:33.570 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-21T23:48:33.734 INFO:teuthology.orchestra.run.smithi031.stdout:34359738372 2024-09-21T23:48:33.734 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-21T23:48:33.735 INFO:teuthology.orchestra.run.smithi031.stdout:34359738371 2024-09-21T23:48:33.735 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-21T23:48:33.736 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2024-09-21T23:48:33.736 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-21T23:48:34.073 INFO:teuthology.orchestra.run.smithi031.stdout:34359738370 2024-09-21T23:48:34.075 INFO:teuthology.orchestra.run.smithi031.stdout:34359738371 2024-09-21T23:48:34.086 INFO:teuthology.orchestra.run.smithi031.stdout:34359738372 2024-09-21T23:48:34.094 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.1 2024-09-21T23:48:34.096 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.0 2024-09-21T23:48:34.106 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.2 2024-09-21T23:48:35.094 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-21T23:48:35.097 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-21T23:48:35.107 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-21T23:48:35.413 INFO:teuthology.orchestra.run.smithi031.stdout:34359738370 2024-09-21T23:48:35.430 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.1 2024-09-21T23:48:35.435 INFO:teuthology.orchestra.run.smithi031.stdout:34359738372 2024-09-21T23:48:35.448 INFO:teuthology.orchestra.run.smithi031.stdout:34359738371 2024-09-21T23:48:35.452 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.2 2024-09-21T23:48:35.465 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.0 2024-09-21T23:48:36.431 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-21T23:48:36.453 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-21T23:48:36.466 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-21T23:48:36.777 INFO:teuthology.orchestra.run.smithi031.stdout:34359738372 2024-09-21T23:48:36.781 INFO:teuthology.orchestra.run.smithi031.stdout:34359738374 2024-09-21T23:48:36.798 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738372 for osd.1 2024-09-21T23:48:36.799 DEBUG:teuthology.parallel:result is None 2024-09-21T23:48:36.801 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738374 for osd.2 2024-09-21T23:48:36.802 DEBUG:teuthology.parallel:result is None 2024-09-21T23:48:36.807 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2024-09-21T23:48:36.824 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738373 for osd.0 2024-09-21T23:48:36.824 DEBUG:teuthology.parallel:result is None 2024-09-21T23:48:36.824 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-21T23:48:36.824 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-21T23:48:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:37.159 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:48:37.175 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-21T23:48:36.553732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82656,"kb_used_data":1552,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032864,"statfs":{"total":289910292480,"available":289825652736,"internally_reserved":0,"allocated":1589248,"data_stored":1295959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":82963792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.070075"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:48:32.523095+0000","last_change":"2024-09-21T23:48:32.523721+0000","last_active":"2024-09-21T23:48:32.523095+0000","last_peered":"2024-09-21T23:48:32.523095+0000","last_clean":"2024-09-21T23:48:32.523095+0000","last_became_active":"2024-09-21T23:48:30.498942+0000","last_became_peered":"2024-09-21T23:48:30.498942+0000","last_unstale":"2024-09-21T23:48:32.523095+0000","last_undegraded":"2024-09-21T23:48:32.523095+0000","last_fullsized":"2024-09-21T23:48:32.523095+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:55:17.700760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019891520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:48:32.522633+0000","last_change":"2024-09-21T23:48:32.522974+0000","last_active":"2024-09-21T23:48:32.522633+0000","last_peered":"2024-09-21T23:48:32.522633+0000","last_clean":"2024-09-21T23:48:32.522633+0000","last_became_active":"2024-09-21T23:48:30.499944+0000","last_became_peered":"2024-09-21T23:48:30.499944+0000","last_unstale":"2024-09-21T23:48:32.522633+0000","last_undegraded":"2024-09-21T23:48:32.522633+0000","last_fullsized":"2024-09-21T23:48:32.522633+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:40:33.520270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016043909999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:48:32.521946+0000","last_change":"2024-09-21T23:48:32.522522+0000","last_active":"2024-09-21T23:48:32.521946+0000","last_peered":"2024-09-21T23:48:32.521946+0000","last_clean":"2024-09-21T23:48:32.521946+0000","last_became_active":"2024-09-21T23:48:30.498992+0000","last_became_peered":"2024-09-21T23:48:30.498992+0000","last_unstale":"2024-09-21T23:48:32.521946+0000","last_undegraded":"2024-09-21T23:48:32.521946+0000","last_fullsized":"2024-09-21T23:48:32.521946+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:30:17.385732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001283916,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:48:32.521717+0000","last_change":"2024-09-21T23:48:32.522399+0000","last_active":"2024-09-21T23:48:32.521717+0000","last_peered":"2024-09-21T23:48:32.521717+0000","last_clean":"2024-09-21T23:48:32.521717+0000","last_became_active":"2024-09-21T23:48:30.499210+0000","last_became_peered":"2024-09-21T23:48:30.499210+0000","last_unstale":"2024-09-21T23:48:32.521717+0000","last_undegraded":"2024-09-21T23:48:32.521717+0000","last_fullsized":"2024-09-21T23:48:32.521717+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:38:47.969858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012027369999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:48:32.518542+0000","last_change":"2024-09-21T23:48:32.518542+0000","last_active":"2024-09-21T23:48:32.518542+0000","last_peered":"2024-09-21T23:48:32.518542+0000","last_clean":"2024-09-21T23:48:32.518542+0000","last_became_active":"2024-09-21T23:48:30.498643+0000","last_became_peered":"2024-09-21T23:48:30.498643+0000","last_unstale":"2024-09-21T23:48:32.518542+0000","last_undegraded":"2024-09-21T23:48:32.518542+0000","last_fullsized":"2024-09-21T23:48:32.518542+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:45:35.695733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019200619999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:48:32.512905+0000","last_change":"2024-09-21T23:48:32.513550+0000","last_active":"2024-09-21T23:48:32.512905+0000","last_peered":"2024-09-21T23:48:32.512905+0000","last_clean":"2024-09-21T23:48:32.512905+0000","last_became_active":"2024-09-21T23:48:30.500705+0000","last_became_peered":"2024-09-21T23:48:30.500705+0000","last_unstale":"2024-09-21T23:48:32.512905+0000","last_undegraded":"2024-09-21T23:48:32.512905+0000","last_fullsized":"2024-09-21T23:48:32.512905+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:21:01.132226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096276900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:48:32.512820+0000","last_change":"2024-09-21T23:48:32.513385+0000","last_active":"2024-09-21T23:48:32.512820+0000","last_peered":"2024-09-21T23:48:32.512820+0000","last_clean":"2024-09-21T23:48:32.512820+0000","last_became_active":"2024-09-21T23:48:30.501645+0000","last_became_peered":"2024-09-21T23:48:30.501645+0000","last_unstale":"2024-09-21T23:48:32.512820+0000","last_undegraded":"2024-09-21T23:48:32.512820+0000","last_fullsized":"2024-09-21T23:48:32.512820+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:12:39.920716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093488900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:48:32.522782+0000","last_change":"2024-09-21T23:48:32.523316+0000","last_active":"2024-09-21T23:48:32.522782+0000","last_peered":"2024-09-21T23:48:32.522782+0000","last_clean":"2024-09-21T23:48:32.522782+0000","last_became_active":"2024-09-21T23:48:30.501010+0000","last_became_peered":"2024-09-21T23:48:30.501010+0000","last_unstale":"2024-09-21T23:48:32.522782+0000","last_undegraded":"2024-09-21T23:48:32.522782+0000","last_fullsized":"2024-09-21T23:48:32.522782+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:40:07.811058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001659017,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'97","reported_seq":147,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:48:32.521089+0000","last_change":"2024-09-21T23:48:26.479535+0000","last_active":"2024-09-21T23:48:32.521089+0000","last_peered":"2024-09-21T23:48:32.521089+0000","last_clean":"2024-09-21T23:48:32.521089+0000","last_became_active":"2024-09-21T23:48:26.475455+0000","last_became_peered":"2024-09-21T23:48:26.475455+0000","last_unstale":"2024-09-21T23:48:32.521089+0000","last_undegraded":"2024-09-21T23:48:32.521089+0000","last_fullsized":"2024-09-21T23:48:32.521089+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:25.462486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:25.462486+0000","last_clean_scrub_stamp":"2024-09-21T23:48:25.462486+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T04:57:09.281011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":38395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:48:37.176 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-21T23:48:37.471 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:37.471 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-21T23:48:37.487 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-21T23:48:36.553732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82656,"kb_used_data":1552,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032864,"statfs":{"total":289910292480,"available":289825652736,"internally_reserved":0,"allocated":1589248,"data_stored":1295959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":82963792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.070075"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:48:32.523095+0000","last_change":"2024-09-21T23:48:32.523721+0000","last_active":"2024-09-21T23:48:32.523095+0000","last_peered":"2024-09-21T23:48:32.523095+0000","last_clean":"2024-09-21T23:48:32.523095+0000","last_became_active":"2024-09-21T23:48:30.498942+0000","last_became_peered":"2024-09-21T23:48:30.498942+0000","last_unstale":"2024-09-21T23:48:32.523095+0000","last_undegraded":"2024-09-21T23:48:32.523095+0000","last_fullsized":"2024-09-21T23:48:32.523095+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:55:17.700760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019891520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:48:32.522633+0000","last_change":"2024-09-21T23:48:32.522974+0000","last_active":"2024-09-21T23:48:32.522633+0000","last_peered":"2024-09-21T23:48:32.522633+0000","last_clean":"2024-09-21T23:48:32.522633+0000","last_became_active":"2024-09-21T23:48:30.499944+0000","last_became_peered":"2024-09-21T23:48:30.499944+0000","last_unstale":"2024-09-21T23:48:32.522633+0000","last_undegraded":"2024-09-21T23:48:32.522633+0000","last_fullsized":"2024-09-21T23:48:32.522633+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:40:33.520270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016043909999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:48:32.521946+0000","last_change":"2024-09-21T23:48:32.522522+0000","last_active":"2024-09-21T23:48:32.521946+0000","last_peered":"2024-09-21T23:48:32.521946+0000","last_clean":"2024-09-21T23:48:32.521946+0000","last_became_active":"2024-09-21T23:48:30.498992+0000","last_became_peered":"2024-09-21T23:48:30.498992+0000","last_unstale":"2024-09-21T23:48:32.521946+0000","last_undegraded":"2024-09-21T23:48:32.521946+0000","last_fullsized":"2024-09-21T23:48:32.521946+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:30:17.385732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001283916,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:48:32.521717+0000","last_change":"2024-09-21T23:48:32.522399+0000","last_active":"2024-09-21T23:48:32.521717+0000","last_peered":"2024-09-21T23:48:32.521717+0000","last_clean":"2024-09-21T23:48:32.521717+0000","last_became_active":"2024-09-21T23:48:30.499210+0000","last_became_peered":"2024-09-21T23:48:30.499210+0000","last_unstale":"2024-09-21T23:48:32.521717+0000","last_undegraded":"2024-09-21T23:48:32.521717+0000","last_fullsized":"2024-09-21T23:48:32.521717+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:38:47.969858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012027369999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:48:32.518542+0000","last_change":"2024-09-21T23:48:32.518542+0000","last_active":"2024-09-21T23:48:32.518542+0000","last_peered":"2024-09-21T23:48:32.518542+0000","last_clean":"2024-09-21T23:48:32.518542+0000","last_became_active":"2024-09-21T23:48:30.498643+0000","last_became_peered":"2024-09-21T23:48:30.498643+0000","last_unstale":"2024-09-21T23:48:32.518542+0000","last_undegraded":"2024-09-21T23:48:32.518542+0000","last_fullsized":"2024-09-21T23:48:32.518542+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:45:35.695733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019200619999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:48:32.512905+0000","last_change":"2024-09-21T23:48:32.513550+0000","last_active":"2024-09-21T23:48:32.512905+0000","last_peered":"2024-09-21T23:48:32.512905+0000","last_clean":"2024-09-21T23:48:32.512905+0000","last_became_active":"2024-09-21T23:48:30.500705+0000","last_became_peered":"2024-09-21T23:48:30.500705+0000","last_unstale":"2024-09-21T23:48:32.512905+0000","last_undegraded":"2024-09-21T23:48:32.512905+0000","last_fullsized":"2024-09-21T23:48:32.512905+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:21:01.132226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096276900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:48:32.512820+0000","last_change":"2024-09-21T23:48:32.513385+0000","last_active":"2024-09-21T23:48:32.512820+0000","last_peered":"2024-09-21T23:48:32.512820+0000","last_clean":"2024-09-21T23:48:32.512820+0000","last_became_active":"2024-09-21T23:48:30.501645+0000","last_became_peered":"2024-09-21T23:48:30.501645+0000","last_unstale":"2024-09-21T23:48:32.512820+0000","last_undegraded":"2024-09-21T23:48:32.512820+0000","last_fullsized":"2024-09-21T23:48:32.512820+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:12:39.920716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093488900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:48:32.522782+0000","last_change":"2024-09-21T23:48:32.523316+0000","last_active":"2024-09-21T23:48:32.522782+0000","last_peered":"2024-09-21T23:48:32.522782+0000","last_clean":"2024-09-21T23:48:32.522782+0000","last_became_active":"2024-09-21T23:48:30.501010+0000","last_became_peered":"2024-09-21T23:48:30.501010+0000","last_unstale":"2024-09-21T23:48:32.522782+0000","last_undegraded":"2024-09-21T23:48:32.522782+0000","last_fullsized":"2024-09-21T23:48:32.522782+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:40:07.811058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001659017,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'97","reported_seq":147,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:48:32.521089+0000","last_change":"2024-09-21T23:48:26.479535+0000","last_active":"2024-09-21T23:48:32.521089+0000","last_peered":"2024-09-21T23:48:32.521089+0000","last_clean":"2024-09-21T23:48:32.521089+0000","last_became_active":"2024-09-21T23:48:26.475455+0000","last_became_peered":"2024-09-21T23:48:26.475455+0000","last_unstale":"2024-09-21T23:48:32.521089+0000","last_undegraded":"2024-09-21T23:48:32.521089+0000","last_fullsized":"2024-09-21T23:48:32.521089+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:25.462486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:25.462486+0000","last_clean_scrub_stamp":"2024-09-21T23:48:25.462486+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T04:57:09.281011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":38395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:48:37.488 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-21T23:48:37.488 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-21T23:48:37.488 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-21T23:48:37.488 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-21T23:48:37.810 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:37.810 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T23:48:37.827 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-21T23:48:37.827 INFO:teuthology.run_tasks:Running task ceph... 2024-09-21T23:48:37.842 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-21T23:48:37.842 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-21T23:48:37.843 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914544', 'branch': 'squid', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/compare-mirror-images-krbd}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7914544', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}], 'name': 'teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}, 'extra_system_packages': ['pv']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 1478, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'sleep_before_teardown': 0, 'subset': '24/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'targets': {'smithi031.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDhw5lLtglvs+U6XQ4fvivqzVMXu4Kh9MFA5QdoAONywSQ1nIxXy+vROrP/azpMzYiv2d39Yuw9pTD29gN4jbrI=', 'smithi060.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNGh2LIWkdpywhXgv5VVuZObANPS13LaR+dESgkBDuxezy2skEuA6fzULv3le0MNVsP4tfY5ZsadoZw+oPrjpgs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'extra_system_packages': ['pv']}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/compare_mirror_images.sh']}, 'env': {'RBD_DEVICE_TYPE': 'krbd'}, 'timeout': '3h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-21T23:48:37.843 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-21T23:48:37.845 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-21T23:48:37.852 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T23:48:37.890 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T23:48:37.908 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:37.908 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T23:48:37.955 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T23:48:37.955 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 2024-09-21T23:48:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T23:48:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T23:48:38.007 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 891 Links: 1 2024-09-21T23:48:38.007 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T23:48:38.007 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-21 23:42:23.896596730 +0000 2024-09-21T23:48:38.007 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-21 23:42:23.644598980 +0000 2024-09-21T23:48:38.007 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-21 23:42:23.644598980 +0000 2024-09-21T23:48:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2024-09-21T23:48:38.007 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T23:48:38.062 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-21T23:48:38.062 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-21T23:48:38.062 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000305021 s, 1.7 MB/s 2024-09-21T23:48:38.063 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T23:48:38.112 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 2024-09-21T23:48:38.159 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T23:48:38.159 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T23:48:38.159 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 905 Links: 1 2024-09-21T23:48:38.159 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T23:48:38.159 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-21 23:42:24.360592585 +0000 2024-09-21T23:48:38.159 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-21 23:42:24.112594800 +0000 2024-09-21T23:48:38.159 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-21 23:42:24.112594800 +0000 2024-09-21T23:48:38.159 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2024-09-21T23:48:38.160 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T23:48:38.213 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-21T23:48:38.213 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-21T23:48:38.213 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000209544 s, 2.4 MB/s 2024-09-21T23:48:38.214 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T23:48:38.264 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 2024-09-21T23:48:38.311 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T23:48:38.311 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T23:48:38.311 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 919 Links: 1 2024-09-21T23:48:38.311 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T23:48:38.311 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-21 23:42:24.880587941 +0000 2024-09-21T23:48:38.311 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-21 23:42:24.620590263 +0000 2024-09-21T23:48:38.311 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-21 23:42:24.620590263 +0000 2024-09-21T23:48:38.312 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2024-09-21T23:48:38.312 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T23:48:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-21T23:48:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-21T23:48:38.366 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000318536 s, 1.6 MB/s 2024-09-21T23:48:38.367 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T23:48:38.416 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 2024-09-21T23:48:38.462 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T23:48:38.463 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T23:48:38.463 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 942 Links: 1 2024-09-21T23:48:38.463 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T23:48:38.463 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-21 23:42:25.108585904 +0000 2024-09-21T23:48:38.463 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-21 23:42:25.104585940 +0000 2024-09-21T23:48:38.463 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-21 23:42:25.104585940 +0000 2024-09-21T23:48:38.463 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2024-09-21T23:48:38.463 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T23:48:38.517 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-21T23:48:38.518 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-21T23:48:38.518 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000305895 s, 1.7 MB/s 2024-09-21T23:48:38.519 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T23:48:38.568 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T23:48:38.568 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-21T23:48:38.568 INFO:tasks.ceph:Generating config... 2024-09-21T23:48:38.569 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-21T23:48:38.569 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-21T23:48:38.569 INFO:tasks.ceph:[client] log to stderr = False 2024-09-21T23:48:38.569 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-21T23:48:38.569 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:38.570 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:38.570 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:38.570 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:38.570 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:38.570 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:38.570 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:38.570 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:38.570 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:38.570 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:38.570 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:38.570 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:38.570 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:38.570 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:38.570 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-21T23:48:38.571 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-21T23:48:38.571 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-21T23:48:38.571 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-21T23:48:38.571 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-21T23:48:38.571 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-21T23:48:38.571 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-21T23:48:38.571 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-21T23:48:38.571 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-09-21T23:48:38.571 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-09-21T23:48:38.571 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-21T23:48:38.571 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-21T23:48:38.571 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-21T23:48:38.571 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-21T23:48:38.571 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-21T23:48:38.571 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-21T23:48:38.571 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-21T23:48:38.572 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-21T23:48:38.572 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-21T23:48:38.572 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-21T23:48:38.572 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-21T23:48:38.572 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-21T23:48:38.572 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-21T23:48:38.572 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-21T23:48:38.643 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster2.keyring 2024-09-21T23:48:38.646 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-21T23:48:38.722 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-21T23:48:38.782 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.60')] 2024-09-21T23:48:38.782 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.60', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-21T23:48:38.783 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:38.783 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T23:48:38.831 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.60 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-21T23:48:38.912 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-21T23:48:38.912 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: generated fsid e1eca878-9fe9-437c-9e28-75a288e822bf 2024-09-21T23:48:38.912 INFO:teuthology.orchestra.run.smithi060.stdout:setting min_mon_release = quincy 2024-09-21T23:48:38.913 INFO:teuthology.orchestra.run.smithi060.stdout:epoch 0 2024-09-21T23:48:38.913 INFO:teuthology.orchestra.run.smithi060.stdout:fsid e1eca878-9fe9-437c-9e28-75a288e822bf 2024-09-21T23:48:38.913 INFO:teuthology.orchestra.run.smithi060.stdout:last_changed 2024-09-21T23:48:38.915194+0000 2024-09-21T23:48:38.913 INFO:teuthology.orchestra.run.smithi060.stdout:created 2024-09-21T23:48:38.915194+0000 2024-09-21T23:48:38.913 INFO:teuthology.orchestra.run.smithi060.stdout:min_mon_release 17 (quincy) 2024-09-21T23:48:38.913 INFO:teuthology.orchestra.run.smithi060.stdout:election_strategy: 1 2024-09-21T23:48:38.913 INFO:teuthology.orchestra.run.smithi060.stdout:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-09-21T23:48:38.914 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-21T23:48:38.915 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T23:48:38.963 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID e1eca878-9fe9-437c-9e28-75a288e822bf... 2024-09-21T23:48:38.964 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-21T23:48:38.967 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-21T23:48:39.009 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2024-09-21T23:48:39.009 INFO:teuthology.orchestra.run.smithi031.stdout: chdir = "" 2024-09-21T23:48:39.009 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T23:48:39.009 INFO:teuthology.orchestra.run.smithi031.stdout: auth supported = cephx 2024-09-21T23:48:39.009 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.009 INFO:teuthology.orchestra.run.smithi031.stdout: filestore xattr use omap = true 2024-09-21T23:48:39.009 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.009 INFO:teuthology.orchestra.run.smithi031.stdout: mon clock drift allowed = 1.000 2024-09-21T23:48:39.010 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.010 INFO:teuthology.orchestra.run.smithi031.stdout: osd crush chooseleaf type = 0 2024-09-21T23:48:39.010 INFO:teuthology.orchestra.run.smithi031.stdout: auth debug = true 2024-09-21T23:48:39.010 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.010 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on old message = true 2024-09-21T23:48:39.010 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on bug = true 2024-09-21T23:48:39.010 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.010 INFO:teuthology.orchestra.run.smithi031.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T23:48:39.010 INFO:teuthology.orchestra.run.smithi031.stdout: mon pg warn max object skew = 0 2024-09-21T23:48:39.010 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.011 INFO:teuthology.orchestra.run.smithi031.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T23:48:39.011 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T23:48:39.011 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.011 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default size = 2 2024-09-21T23:48:39.011 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.011 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow primary affinity = true 2024-09-21T23:48:39.011 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow pg remap = true 2024-09-21T23:48:39.011 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on legacy crush tunables = false 2024-09-21T23:48:39.011 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on crush straw calc version zero = false 2024-09-21T23:48:39.011 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on no sortbitwise = false 2024-09-21T23:48:39.011 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on osd down out interval zero = false 2024-09-21T23:48:39.011 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on too few osds = false 2024-09-21T23:48:39.011 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T23:48:39.011 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T23:48:39.012 INFO:teuthology.orchestra.run.smithi031.stdout: mon_allow_pool_size_one = true 2024-09-21T23:48:39.012 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.012 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T23:48:39.012 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.012 INFO:teuthology.orchestra.run.smithi031.stdout: osd default data pool replay window = 5 2024-09-21T23:48:39.012 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.012 INFO:teuthology.orchestra.run.smithi031.stdout: mon allow pool delete = true 2024-09-21T23:48:39.012 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.012 INFO:teuthology.orchestra.run.smithi031.stdout: mon cluster log file level = debug 2024-09-21T23:48:39.012 INFO:teuthology.orchestra.run.smithi031.stdout: debug asserts on shutdown = true 2024-09-21T23:48:39.012 INFO:teuthology.orchestra.run.smithi031.stdout: mon health detail to clog = false 2024-09-21T23:48:39.012 INFO:teuthology.orchestra.run.smithi031.stdout: mon host = 172.21.15.60 2024-09-21T23:48:39.012 INFO:teuthology.orchestra.run.smithi031.stdout: mon client directed command retry = 5 2024-09-21T23:48:39.012 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on pool no app = False 2024-09-21T23:48:39.012 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject socket failures = 5000 2024-09-21T23:48:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = e1eca878-9fe9-437c-9e28-75a288e822bf 2024-09-21T23:48:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.013 INFO:teuthology.orchestra.run.smithi031.stdout:[osd] 2024-09-21T23:48:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: osd journal size = 100 2024-09-21T23:48:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub load threshold = 5.0 2024-09-21T23:48:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub max interval = 600 2024-09-21T23:48:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock profile = high_recovery_ops 2024-09-21T23:48:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: osd recover clone overlap = true 2024-09-21T23:48:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: osd recovery max chunk = 1048576 2024-09-21T23:48:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug shutdown = true 2024-09-21T23:48:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug op order = true 2024-09-21T23:48:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify stray on activate = true 2024-09-21T23:48:39.013 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug trim objects = true 2024-09-21T23:48:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: osd open classes on start = true 2024-09-21T23:48:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug pg log writeout = true 2024-09-21T23:48:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: osd deep scrub update digest min age = 30 2024-09-21T23:48:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: osd map max advance = 10 2024-09-21T23:48:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: journal zero on create = true 2024-09-21T23:48:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: filestore ondisk finisher threads = 3 2024-09-21T23:48:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: filestore apply finisher threads = 3 2024-09-21T23:48:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: bdev debug aio = true 2024-09-21T23:48:39.014 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug misdirected ops = true 2024-09-21T23:48:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: bluefs allocator = hybrid 2024-09-21T23:48:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore allocator = hybrid 2024-09-21T23:48:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore block size = 96636764160 2024-09-21T23:48:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore fsck on mount = True 2024-09-21T23:48:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluefs = 1/20 2024-09-21T23:48:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluestore = 1/20 2024-09-21T23:48:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-09-21T23:48:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 20 2024-09-21T23:48:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: debug rocksdb = 4/10 2024-09-21T23:48:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T23:48:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd full ratio = 0.9 2024-09-21T23:48:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd nearfull ratio = 0.8 2024-09-21T23:48:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: osd failsafe full ratio = 0.95 2024-09-21T23:48:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: osd objectstore = bluestore 2024-09-21T23:48:39.015 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.016 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr] 2024-09-21T23:48:39.016 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-09-21T23:48:39.016 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 20 2024-09-21T23:48:39.016 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2024-09-21T23:48:39.016 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2024-09-21T23:48:39.016 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2024-09-21T23:48:39.016 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2024-09-21T23:48:39.016 INFO:teuthology.orchestra.run.smithi031.stdout: mgr/telemetry/nag = false 2024-09-21T23:48:39.016 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.016 INFO:teuthology.orchestra.run.smithi031.stdout:[mon] 2024-09-21T23:48:39.016 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-09-21T23:48:39.016 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2024-09-21T23:48:39.016 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 20 2024-09-21T23:48:39.016 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2024-09-21T23:48:39.016 INFO:teuthology.orchestra.run.smithi031.stdout: mon data avail warn = 5 2024-09-21T23:48:39.016 INFO:teuthology.orchestra.run.smithi031.stdout: mon mgr mkfs grace = 240 2024-09-21T23:48:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2024-09-21T23:48:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd reporter subtree level = osd 2024-09-21T23:48:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd prime pg temp = true 2024-09-21T23:48:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2024-09-21T23:48:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T23:48:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T23:48:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T23:48:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T23:48:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T23:48:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T23:48:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.017 INFO:teuthology.orchestra.run.smithi031.stdout: # 1m isn't quite enough 2024-09-21T23:48:39.018 INFO:teuthology.orchestra.run.smithi031.stdout: mon_down_mkfs_grace = 2m 2024-09-21T23:48:39.018 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.018 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_filestore_osds = false 2024-09-21T23:48:39.018 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-21T23:48:39.018 INFO:teuthology.orchestra.run.smithi031.stdout:[client] 2024-09-21T23:48:39.018 INFO:teuthology.orchestra.run.smithi031.stdout: rgw cache enabled = true 2024-09-21T23:48:39.018 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable ops log = true 2024-09-21T23:48:39.018 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable usage log = true 2024-09-21T23:48:39.018 INFO:teuthology.orchestra.run.smithi031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T23:48:39.018 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T23:48:39.018 INFO:teuthology.orchestra.run.smithi031.stdout: debug rbd = 20 2024-09-21T23:48:39.018 INFO:teuthology.orchestra.run.smithi031.stdout: debug rbd_mirror = 15 2024-09-21T23:48:39.018 INFO:teuthology.orchestra.run.smithi031.stdout: log to stderr = False 2024-09-21T23:48:39.019 INFO:teuthology.orchestra.run.smithi031.stdout: rbd default features = 125 2024-09-21T23:48:39.019 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.a] 2024-09-21T23:48:39.019 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.0] 2024-09-21T23:48:39.019 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:39.019 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:39.019 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.1] 2024-09-21T23:48:39.019 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:39.019 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:39.019 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.2] 2024-09-21T23:48:39.019 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:39.019 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:39.019 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.3] 2024-09-21T23:48:39.019 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:39.019 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:39.019 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.4] 2024-09-21T23:48:39.020 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:39.020 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:39.020 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.5] 2024-09-21T23:48:39.020 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:39.020 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:39.020 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.6] 2024-09-21T23:48:39.020 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:39.020 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:39.038 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2024-09-21T23:48:39.038 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2024-09-21T23:48:39.038 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T23:48:39.038 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2024-09-21T23:48:39.038 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.038 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2024-09-21T23:48:39.038 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.038 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2024-09-21T23:48:39.038 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.038 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2024-09-21T23:48:39.038 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2024-09-21T23:48:39.039 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.039 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2024-09-21T23:48:39.039 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2024-09-21T23:48:39.039 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.039 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T23:48:39.039 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2024-09-21T23:48:39.039 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.039 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T23:48:39.039 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T23:48:39.039 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.039 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2024-09-21T23:48:39.039 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.039 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2024-09-21T23:48:39.039 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2024-09-21T23:48:39.039 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2024-09-21T23:48:39.040 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2024-09-21T23:48:39.040 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2024-09-21T23:48:39.040 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2024-09-21T23:48:39.040 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2024-09-21T23:48:39.040 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T23:48:39.040 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T23:48:39.040 INFO:teuthology.orchestra.run.smithi060.stdout: mon_allow_pool_size_one = true 2024-09-21T23:48:39.040 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.040 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T23:48:39.040 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.040 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2024-09-21T23:48:39.040 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.041 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2024-09-21T23:48:39.041 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.041 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2024-09-21T23:48:39.041 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2024-09-21T23:48:39.041 INFO:teuthology.orchestra.run.smithi060.stdout: mon health detail to clog = false 2024-09-21T23:48:39.041 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = 172.21.15.60 2024-09-21T23:48:39.041 INFO:teuthology.orchestra.run.smithi060.stdout: mon client directed command retry = 5 2024-09-21T23:48:39.041 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on pool no app = False 2024-09-21T23:48:39.041 INFO:teuthology.orchestra.run.smithi060.stdout: ms inject socket failures = 5000 2024-09-21T23:48:39.041 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = e1eca878-9fe9-437c-9e28-75a288e822bf 2024-09-21T23:48:39.041 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.042 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2024-09-21T23:48:39.042 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2024-09-21T23:48:39.042 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.042 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2024-09-21T23:48:39.042 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2024-09-21T23:48:39.042 INFO:teuthology.orchestra.run.smithi060.stdout: osd mclock profile = high_recovery_ops 2024-09-21T23:48:39.042 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.042 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2024-09-21T23:48:39.042 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2024-09-21T23:48:39.042 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.042 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2024-09-21T23:48:39.042 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2024-09-21T23:48:39.042 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2024-09-21T23:48:39.042 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug trim objects = true 2024-09-21T23:48:39.042 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.043 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2024-09-21T23:48:39.043 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2024-09-21T23:48:39.043 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.043 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2024-09-21T23:48:39.043 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.043 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2024-09-21T23:48:39.043 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.043 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2024-09-21T23:48:39.043 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.043 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2024-09-21T23:48:39.043 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2024-09-21T23:48:39.043 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.043 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2024-09-21T23:48:39.043 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2024-09-21T23:48:39.043 INFO:teuthology.orchestra.run.smithi060.stdout: bluefs allocator = hybrid 2024-09-21T23:48:39.043 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore allocator = hybrid 2024-09-21T23:48:39.044 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore block size = 96636764160 2024-09-21T23:48:39.044 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore fsck on mount = True 2024-09-21T23:48:39.044 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluefs = 1/20 2024-09-21T23:48:39.044 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluestore = 1/20 2024-09-21T23:48:39.044 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-09-21T23:48:39.044 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 20 2024-09-21T23:48:39.044 INFO:teuthology.orchestra.run.smithi060.stdout: debug rocksdb = 4/10 2024-09-21T23:48:39.044 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T23:48:39.044 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd full ratio = 0.9 2024-09-21T23:48:39.044 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd nearfull ratio = 0.8 2024-09-21T23:48:39.044 INFO:teuthology.orchestra.run.smithi060.stdout: osd failsafe full ratio = 0.95 2024-09-21T23:48:39.044 INFO:teuthology.orchestra.run.smithi060.stdout: osd objectstore = bluestore 2024-09-21T23:48:39.044 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.044 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2024-09-21T23:48:39.044 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-09-21T23:48:39.044 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2024-09-21T23:48:39.044 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-09-21T23:48:39.045 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-09-21T23:48:39.045 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-09-21T23:48:39.045 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-09-21T23:48:39.045 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2024-09-21T23:48:39.045 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.045 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2024-09-21T23:48:39.045 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-09-21T23:48:39.045 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-09-21T23:48:39.045 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2024-09-21T23:48:39.045 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-09-21T23:48:39.045 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2024-09-21T23:48:39.045 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2024-09-21T23:48:39.045 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-09-21T23:48:39.045 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2024-09-21T23:48:39.045 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2024-09-21T23:48:39.046 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-09-21T23:48:39.046 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.046 INFO:teuthology.orchestra.run.smithi060.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T23:48:39.046 INFO:teuthology.orchestra.run.smithi060.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T23:48:39.046 INFO:teuthology.orchestra.run.smithi060.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T23:48:39.046 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.046 INFO:teuthology.orchestra.run.smithi060.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T23:48:39.046 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T23:48:39.046 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T23:48:39.046 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.046 INFO:teuthology.orchestra.run.smithi060.stdout: # 1m isn't quite enough 2024-09-21T23:48:39.046 INFO:teuthology.orchestra.run.smithi060.stdout: mon_down_mkfs_grace = 2m 2024-09-21T23:48:39.046 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.046 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_filestore_osds = false 2024-09-21T23:48:39.047 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:39.047 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2024-09-21T23:48:39.047 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2024-09-21T23:48:39.047 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2024-09-21T23:48:39.047 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2024-09-21T23:48:39.047 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T23:48:39.047 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T23:48:39.047 INFO:teuthology.orchestra.run.smithi060.stdout: debug rbd = 20 2024-09-21T23:48:39.047 INFO:teuthology.orchestra.run.smithi060.stdout: debug rbd_mirror = 15 2024-09-21T23:48:39.047 INFO:teuthology.orchestra.run.smithi060.stdout: log to stderr = False 2024-09-21T23:48:39.047 INFO:teuthology.orchestra.run.smithi060.stdout: rbd default features = 125 2024-09-21T23:48:39.047 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2024-09-21T23:48:39.047 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.0] 2024-09-21T23:48:39.047 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:39.047 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:39.048 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.1] 2024-09-21T23:48:39.048 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:39.048 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:39.048 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.2] 2024-09-21T23:48:39.048 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:39.048 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:39.048 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.3] 2024-09-21T23:48:39.048 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:39.048 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:39.048 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.4] 2024-09-21T23:48:39.048 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:39.048 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:39.048 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.5] 2024-09-21T23:48:39.048 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:39.048 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:39.049 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.6] 2024-09-21T23:48:39.049 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-21T23:48:39.049 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-21T23:48:39.049 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-21T23:48:39.050 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-21T23:48:39.143 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-21T23:48:39.144 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:39.144 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-21T23:48:39.149 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:39.149 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-21T23:48:39.194 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi031.front.sepia.ceph.com 2024-09-21T23:48:39.195 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:48:39.195 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-21T23:48:39.195 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-21T23:48:39.219 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:48:39.219 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-21T23:48:39.266 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:48:39.266 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:39.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-21T23:48:39.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-21T23:48:39.289 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:39.289 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-21T23:48:39.335 INFO:tasks.ceph:Setting up mon nodes... 2024-09-21T23:48:39.335 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-21T23:48:39.335 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-21T23:48:39.381 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-21T23:48:39.383 INFO:tasks.ceph:Setting up mds nodes... 2024-09-21T23:48:39.384 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-21T23:48:39.384 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-21T23:48:39.461 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-21T23:48:39.472 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-21T23:48:39.509 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-21T23:48:39.521 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-21T23:48:39.596 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-21T23:48:39.607 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-21T23:48:39.682 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-21T23:48:39.692 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-21T23:48:39.765 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-21T23:48:39.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-21T23:48:39.855 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-21T23:48:39.865 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-21T23:48:39.939 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-21T23:48:39.950 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-21T23:48:40.028 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-21T23:48:40.039 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-21T23:48:40.120 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-21T23:48:40.132 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-21T23:48:40.132 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-21T23:48:40.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-21T23:48:40.187 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T23:48:40.187 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-21T23:48:40.187 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:48:40.187 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T23:48:40.242 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T23:48:40.242 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T23:48:40.243 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T23:48:40.243 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T23:48:40.243 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T23:48:40.243 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-09-21T23:48:40.243 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T23:48:40.243 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T23:48:40.243 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T23:48:40.243 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T23:48:40.266 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-09-21T23:48:40.268 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-09-21T23:48:40.269 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-21T23:48:40.326 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-21T23:48:40.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T23:48:40.380 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T23:48:40.381 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-21T23:48:40.433 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T23:48:40.433 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-21T23:48:40.433 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:48:40.433 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T23:48:40.487 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T23:48:40.487 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T23:48:40.487 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T23:48:40.487 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T23:48:40.487 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T23:48:40.487 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-09-21T23:48:40.487 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T23:48:40.487 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T23:48:40.487 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T23:48:40.488 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T23:48:40.511 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-09-21T23:48:40.512 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-09-21T23:48:40.512 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-21T23:48:40.570 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-21T23:48:40.623 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T23:48:40.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T23:48:40.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-21T23:48:40.677 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T23:48:40.677 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-21T23:48:40.677 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:48:40.677 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T23:48:40.729 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T23:48:40.730 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T23:48:40.730 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T23:48:40.730 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T23:48:40.730 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T23:48:40.730 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-09-21T23:48:40.730 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T23:48:40.730 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T23:48:40.730 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T23:48:40.730 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T23:48:40.753 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-09-21T23:48:40.755 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-09-21T23:48:40.755 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-21T23:48:40.814 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-21T23:48:40.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T23:48:40.867 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T23:48:40.867 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-21T23:48:40.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:48:40.976+0000 7f51ecc3d7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-21T23:48:40.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:48:40.976+0000 7f51ecc3d7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-21T23:48:40.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:48:40.976+0000 7f51ecc3d7c0 -1 bdev(0x560fa2525800 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2024-09-21T23:48:40.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:48:40.976+0000 7f51ecc3d7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-09-21T23:48:44.103 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-21T23:48:44.116 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-21T23:48:44.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:48:44.196+0000 7fc8bed047c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-21T23:48:44.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:48:44.196+0000 7fc8bed047c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-21T23:48:44.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:48:44.196+0000 7fc8bed047c0 -1 bdev(0x5631e346d800 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2024-09-21T23:48:44.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:48:44.200+0000 7fc8bed047c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-09-21T23:48:47.336 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-21T23:48:47.391 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-21T23:48:47.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:48:47.472+0000 7f350de6e7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-21T23:48:47.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:48:47.472+0000 7f350de6e7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-21T23:48:47.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:48:47.472+0000 7f350de6e7c0 -1 bdev(0x5590ad90d800 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2024-09-21T23:48:47.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:48:47.472+0000 7f350de6e7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-09-21T23:48:50.614 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-21T23:48:50.672 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-21T23:48:50.672 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:48:50.672 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-21T23:48:50.685 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:50.685 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-21T23:48:50.727 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:50.727 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-21T23:48:50.783 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:50.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-21T23:48:50.839 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-21T23:48:50.839 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-21T23:48:50.845 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:50.845 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-21T23:48:50.887 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:50.887 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-21T23:48:50.935 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:50.935 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-21T23:48:50.983 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:50.983 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-21T23:48:51.031 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:51.031 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-21T23:48:51.079 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:51.079 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-21T23:48:51.127 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:51.127 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-21T23:48:51.175 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:51.175 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-21T23:48:51.223 INFO:tasks.ceph:Adding keys to all mons... 2024-09-21T23:48:51.223 DEBUG:teuthology.orchestra.run.smithi060:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-21T23:48:51.277 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.x] 2024-09-21T23:48:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBXW+9mn2fyFhAA3v33u3vr/iw+sxrUC5isMA== 2024-09-21T23:48:51.278 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2024-09-21T23:48:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBYW+9mTOJOOhAAO32kb2NDV4NntJ5r74BGpQ== 2024-09-21T23:48:51.278 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2024-09-21T23:48:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBcW+9miQD3CxAAvAOxAhMhpr3aWs5vUMmMPA== 2024-09-21T23:48:51.278 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2024-09-21T23:48:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBfW+9m26tYHBAA8LvrILGvcNVkTiztzk3/uA== 2024-09-21T23:48:51.278 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2024-09-21T23:48:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBXW+9m1UDAGxAAJhTBx/FODovk+x0Q4Wcv8w== 2024-09-21T23:48:51.278 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror] 2024-09-21T23:48:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBXW+9m0FGLHhAAiYfDrXzwt0ukbgD5+GVVUA== 2024-09-21T23:48:51.278 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.0] 2024-09-21T23:48:51.278 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBXW+9mYuu4IxAA1iOGymSfzQHGEJNa49CcgQ== 2024-09-21T23:48:51.279 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.1] 2024-09-21T23:48:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBXW+9mgAzZKBAAp+tDgHH4phx+4lMhvqe2Tw== 2024-09-21T23:48:51.279 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.2] 2024-09-21T23:48:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBXW+9mp1PGLRAADnmAGQBmFmQ6acQYCdwerQ== 2024-09-21T23:48:51.279 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.3] 2024-09-21T23:48:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBXW+9mc6YjMxAAW0aBNo5H6aXNuDuIBRXBfw== 2024-09-21T23:48:51.279 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.4] 2024-09-21T23:48:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBXW+9mqQUmOBAAjhl1qSrOSF81zHMSpeCh+A== 2024-09-21T23:48:51.279 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.5] 2024-09-21T23:48:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBYW+9mSoLVARAAbKB7H6nAkyK8VseLrgDXWA== 2024-09-21T23:48:51.279 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.6] 2024-09-21T23:48:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBYW+9mmwRbBxAA0WPOQvz+nX3lPohhnABUyA== 2024-09-21T23:48:51.280 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T23:48:51.369 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:48:51.411 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:48:51.504 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T23:48:51.590 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:48:51.682 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:48:51.726 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:48:51.815 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:48:51.859 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:48:51.902 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:48:51.989 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:48:52.079 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:48:52.124 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T23:48:52.211 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-21T23:48:52.211 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-21T23:48:52.224 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-21T23:48:52.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-21T23:48:52.323 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-21T23:48:52.370 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-21T23:48:52.370 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-21T23:48:52.370 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-21T23:48:52.415 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-21T23:48:52.415 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-21T23:48:52.416 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-21T23:48:52.416 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-21T23:48:52.418 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-21T23:48:52.418 INFO:tasks.ceph:Setting crush tunables to default 2024-09-21T23:48:52.419 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-21T23:48:52.642 INFO:teuthology.orchestra.run.smithi060.stderr:adjusted tunables profile to default 2024-09-21T23:48:52.661 INFO:tasks.ceph:check_enable_crimson: False 2024-09-21T23:48:52.661 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-21T23:48:52.661 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:52.661 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-21T23:48:52.673 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:52.673 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-21T23:48:52.725 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:48:52.725 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-21T23:48:52.777 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster cluster2 osd new 3f4c56fb-237d-428d-9dcd-e02e7cb1904b 0 2024-09-21T23:48:52.804 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:52.802+0000 7f2b0996c180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T23:48:53.028 INFO:teuthology.orchestra.run.smithi060.stdout:0 2024-09-21T23:48:53.046 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster cluster2 osd new 6d00fac8-41ee-491e-b0d5-1ebe6d17078b 1 2024-09-21T23:48:53.211 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:53.210+0000 7f2b0996c180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T23:48:53.243 INFO:teuthology.orchestra.run.smithi060.stdout:1 2024-09-21T23:48:53.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster cluster2 osd new 5ed7107f-a3cf-4376-8904-7c93e47acda5 2 2024-09-21T23:48:53.356 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:53.354+0000 7f2b0996c180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T23:48:53.467 INFO:teuthology.orchestra.run.smithi060.stdout:2 2024-09-21T23:48:53.484 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-21T23:48:53.484 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-21T23:48:53.486 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-21T23:48:53.486 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-21T23:48:53.486 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-21T23:48:53.488 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-21T23:48:53.488 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-21T23:48:53.488 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-21T23:48:53.490 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-21T23:48:53.490 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-21T23:48:53.798 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:53.798+0000 7f2b0996c180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T23:48:53.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-21T23:48:53.820+0000 7fe4e259c640 0 --2- 172.21.15.60:0/3519703242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4e414d020 0x7fe4e40b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-21T23:48:53.852 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:53.850+0000 7f2b0996c180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T23:48:53.886 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:48:53.886 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":5,"fsid":"e1eca878-9fe9-437c-9e28-75a288e822bf","created":"2024-09-21T23:48:52.550274+0000","modified":"2024-09-21T23:48:53.468593+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T23:48:53.468593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3f4c56fb-237d-428d-9dcd-e02e7cb1904b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d00fac8-41ee-491e-b0d5-1ebe6d17078b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ed7107f-a3cf-4376-8904-7c93e47acda5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:48:53.902 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:53.902+0000 7f2b0996c180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T23:48:53.904 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-21T23:48:53.904 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-21T23:48:53.980 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:53.978+0000 7f2b0996c180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T23:48:54.032 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:54.030+0000 7f2b0996c180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T23:48:54.108 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:54.106+0000 7f2b0996c180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T23:48:54.161 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:54.162+0000 7f2b0996c180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T23:48:54.206 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-21T23:48:54.307 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:54.306+0000 7f2b0996c180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T23:48:54.361 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:54.358+0000 7f2b0996c180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T23:48:54.398 INFO:teuthology.misc.health.smithi060.stdout: 2024-09-21T23:48:54.398 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":5,"fsid":"e1eca878-9fe9-437c-9e28-75a288e822bf","created":"2024-09-21T23:48:52.550274+0000","modified":"2024-09-21T23:48:53.468593+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T23:48:53.468593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3f4c56fb-237d-428d-9dcd-e02e7cb1904b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d00fac8-41ee-491e-b0d5-1ebe6d17078b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ed7107f-a3cf-4376-8904-7c93e47acda5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:48:54.414 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-21T23:48:54.587 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-21T23:48:54.587 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-21T23:48:54.587 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr: from numpy import show_config as show_numpy_config 2024-09-21T23:48:54.595 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:54.594+0000 7f2b0996c180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T23:48:54.600 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-21T23:48:54.600+0000 7fab2faa77c0 -1 Falling back to public interface 2024-09-21T23:48:54.611 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-21T23:48:54.612+0000 7fc7c725c7c0 -1 Falling back to public interface 2024-09-21T23:48:54.736 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:54.734+0000 7f2b0996c180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T23:48:54.850 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:54.850+0000 7f2b0996c180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T23:48:54.868 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-21T23:48:54.868+0000 7f84a59d67c0 -1 Falling back to public interface 2024-09-21T23:48:54.915 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:54.914+0000 7f2b0996c180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T23:48:55.055 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:55.054+0000 7f2b0996c180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T23:48:55.557 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:55.554+0000 7f2b0996c180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T23:48:55.634 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:55.634+0000 7f2b0996c180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T23:48:55.785 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:55.782+0000 7f2b0996c180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T23:48:55.861 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:55.858+0000 7f2b0996c180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T23:48:56.159 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:56.158+0000 7f2b0996c180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T23:48:56.216 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:56.214+0000 7f2b0996c180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T23:48:58.245 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2024-09-21T23:48:58.242+0000 7f2ab05a0640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-21T23:48:58.536 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-21T23:48:58.536+0000 7fab2faa77c0 -1 osd.0 0 log_to_monitors true 2024-09-21T23:48:58.553 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-21T23:48:58.552+0000 7fc7c725c7c0 -1 osd.1 0 log_to_monitors true 2024-09-21T23:48:58.705 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-21T23:48:58.704+0000 7f84a59d67c0 -1 osd.2 0 log_to_monitors true 2024-09-21T23:49:00.717 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-21T23:49:01.165 INFO:teuthology.misc.health.smithi060.stdout: 2024-09-21T23:49:01.165 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":7,"fsid":"e1eca878-9fe9-437c-9e28-75a288e822bf","created":"2024-09-21T23:48:52.550274+0000","modified":"2024-09-21T23:49:00.304205+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T23:48:53.468593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3f4c56fb-237d-428d-9dcd-e02e7cb1904b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d00fac8-41ee-491e-b0d5-1ebe6d17078b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ed7107f-a3cf-4376-8904-7c93e47acda5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:49:01.191 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-21T23:49:01.675 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-21T23:49:01.676+0000 7f84a197f640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T23:49:01.680 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-21T23:49:01.680+0000 7fab2ba50640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T23:49:01.695 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-21T23:49:01.696+0000 7f84997b4640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T23:49:01.702 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-21T23:49:01.704+0000 7fc7c3a17640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T23:49:01.707 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-21T23:49:01.708+0000 7fab23885640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T23:49:01.722 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-21T23:49:01.724+0000 7fc7bb03a640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T23:49:07.495 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-21T23:49:07.801 INFO:teuthology.misc.health.smithi060.stdout: 2024-09-21T23:49:07.801 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":11,"fsid":"e1eca878-9fe9-437c-9e28-75a288e822bf","created":"2024-09-21T23:48:52.550274+0000","modified":"2024-09-21T23:49:06.338378+0000","last_up_change":"2024-09-21T23:49:02.706803+0000","last_in_change":"2024-09-21T23:48:53.468593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T23:49:04.326544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f4c56fb-237d-428d-9dcd-e02e7cb1904b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1391603771},{"type":"v1","addr":"172.21.15.60:6801","nonce":1391603771}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1391603771},{"type":"v1","addr":"172.21.15.60:6803","nonce":1391603771}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1391603771},{"type":"v1","addr":"172.21.15.60:6807","nonce":1391603771}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1391603771},{"type":"v1","addr":"172.21.15.60:6805","nonce":1391603771}]},"public_addr":"172.21.15.60:6801/1391603771","cluster_addr":"172.21.15.60:6803/1391603771","heartbeat_back_addr":"172.21.15.60:6807/1391603771","heartbeat_front_addr":"172.21.15.60:6805/1391603771","state":["exists","up"]},{"osd":1,"uuid":"6d00fac8-41ee-491e-b0d5-1ebe6d17078b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3160429338},{"type":"v1","addr":"172.21.15.60:6809","nonce":3160429338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3160429338},{"type":"v1","addr":"172.21.15.60:6811","nonce":3160429338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3160429338},{"type":"v1","addr":"172.21.15.60:6815","nonce":3160429338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3160429338},{"type":"v1","addr":"172.21.15.60:6813","nonce":3160429338}]},"public_addr":"172.21.15.60:6809/3160429338","cluster_addr":"172.21.15.60:6811/3160429338","heartbeat_back_addr":"172.21.15.60:6815/3160429338","heartbeat_front_addr":"172.21.15.60:6813/3160429338","state":["exists","up"]},{"osd":2,"uuid":"5ed7107f-a3cf-4376-8904-7c93e47acda5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":2394404319},{"type":"v1","addr":"172.21.15.60:6817","nonce":2394404319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":2394404319},{"type":"v1","addr":"172.21.15.60:6819","nonce":2394404319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":2394404319},{"type":"v1","addr":"172.21.15.60:6823","nonce":2394404319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2394404319},{"type":"v1","addr":"172.21.15.60:6821","nonce":2394404319}]},"public_addr":"172.21.15.60:6817/2394404319","cluster_addr":"172.21.15.60:6819/2394404319","heartbeat_back_addr":"172.21.15.60:6823/2394404319","heartbeat_front_addr":"172.21.15.60:6821/2394404319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:59.583845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:59.538461+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:59.727579+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:49:07.819 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-21T23:49:07.820 INFO:tasks.ceph:Creating RBD pool 2024-09-21T23:49:07.820 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-21T23:49:08.348 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'rbd' created 2024-09-21T23:49:08.374 DEBUG:teuthology.orchestra.run.smithi060:> rbd --cluster cluster2 pool init rbd 2024-09-21T23:49:11.383 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-21T23:49:11.384 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-21T23:49:11.384 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-21T23:49:11.384 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-21T23:49:11.384 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-21T23:49:11.742 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:49:11.760 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3789814444},{"type":"v1","addr":"172.21.15.31:6827","nonce":3789814444}]},"active_addr":"172.21.15.31:6827/3789814444","active_change":"2024-09-21T23:48:56.238163+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":3537217417}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":987777682}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":1454485856}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2666575276}]}]} 2024-09-21T23:49:11.762 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-21T23:49:11.762 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-21T23:49:11.762 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-21T23:49:12.060 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:49:12.061 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":15,"fsid":"e1eca878-9fe9-437c-9e28-75a288e822bf","created":"2024-09-21T23:48:52.550274+0000","modified":"2024-09-21T23:49:11.361625+0000","last_up_change":"2024-09-21T23:49:02.706803+0000","last_in_change":"2024-09-21T23:48:53.468593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T23:49:04.326544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T23:49:08.167148+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f4c56fb-237d-428d-9dcd-e02e7cb1904b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1391603771},{"type":"v1","addr":"172.21.15.60:6801","nonce":1391603771}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1391603771},{"type":"v1","addr":"172.21.15.60:6803","nonce":1391603771}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1391603771},{"type":"v1","addr":"172.21.15.60:6807","nonce":1391603771}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1391603771},{"type":"v1","addr":"172.21.15.60:6805","nonce":1391603771}]},"public_addr":"172.21.15.60:6801/1391603771","cluster_addr":"172.21.15.60:6803/1391603771","heartbeat_back_addr":"172.21.15.60:6807/1391603771","heartbeat_front_addr":"172.21.15.60:6805/1391603771","state":["exists","up"]},{"osd":1,"uuid":"6d00fac8-41ee-491e-b0d5-1ebe6d17078b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3160429338},{"type":"v1","addr":"172.21.15.60:6809","nonce":3160429338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3160429338},{"type":"v1","addr":"172.21.15.60:6811","nonce":3160429338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3160429338},{"type":"v1","addr":"172.21.15.60:6815","nonce":3160429338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3160429338},{"type":"v1","addr":"172.21.15.60:6813","nonce":3160429338}]},"public_addr":"172.21.15.60:6809/3160429338","cluster_addr":"172.21.15.60:6811/3160429338","heartbeat_back_addr":"172.21.15.60:6815/3160429338","heartbeat_front_addr":"172.21.15.60:6813/3160429338","state":["exists","up"]},{"osd":2,"uuid":"5ed7107f-a3cf-4376-8904-7c93e47acda5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":2394404319},{"type":"v1","addr":"172.21.15.60:6817","nonce":2394404319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":2394404319},{"type":"v1","addr":"172.21.15.60:6819","nonce":2394404319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":2394404319},{"type":"v1","addr":"172.21.15.60:6823","nonce":2394404319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2394404319},{"type":"v1","addr":"172.21.15.60:6821","nonce":2394404319}]},"public_addr":"172.21.15.60:6817/2394404319","cluster_addr":"172.21.15.60:6819/2394404319","heartbeat_back_addr":"172.21.15.60:6823/2394404319","heartbeat_front_addr":"172.21.15.60:6821/2394404319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:59.583845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:59.538461+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:59.727579+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:49:12.079 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-21T23:49:12.080 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-21T23:49:12.378 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:49:12.378 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":15,"fsid":"e1eca878-9fe9-437c-9e28-75a288e822bf","created":"2024-09-21T23:48:52.550274+0000","modified":"2024-09-21T23:49:11.361625+0000","last_up_change":"2024-09-21T23:49:02.706803+0000","last_in_change":"2024-09-21T23:48:53.468593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T23:49:04.326544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T23:49:08.167148+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f4c56fb-237d-428d-9dcd-e02e7cb1904b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1391603771},{"type":"v1","addr":"172.21.15.60:6801","nonce":1391603771}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1391603771},{"type":"v1","addr":"172.21.15.60:6803","nonce":1391603771}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1391603771},{"type":"v1","addr":"172.21.15.60:6807","nonce":1391603771}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1391603771},{"type":"v1","addr":"172.21.15.60:6805","nonce":1391603771}]},"public_addr":"172.21.15.60:6801/1391603771","cluster_addr":"172.21.15.60:6803/1391603771","heartbeat_back_addr":"172.21.15.60:6807/1391603771","heartbeat_front_addr":"172.21.15.60:6805/1391603771","state":["exists","up"]},{"osd":1,"uuid":"6d00fac8-41ee-491e-b0d5-1ebe6d17078b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3160429338},{"type":"v1","addr":"172.21.15.60:6809","nonce":3160429338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3160429338},{"type":"v1","addr":"172.21.15.60:6811","nonce":3160429338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3160429338},{"type":"v1","addr":"172.21.15.60:6815","nonce":3160429338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3160429338},{"type":"v1","addr":"172.21.15.60:6813","nonce":3160429338}]},"public_addr":"172.21.15.60:6809/3160429338","cluster_addr":"172.21.15.60:6811/3160429338","heartbeat_back_addr":"172.21.15.60:6815/3160429338","heartbeat_front_addr":"172.21.15.60:6813/3160429338","state":["exists","up"]},{"osd":2,"uuid":"5ed7107f-a3cf-4376-8904-7c93e47acda5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":2394404319},{"type":"v1","addr":"172.21.15.60:6817","nonce":2394404319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":2394404319},{"type":"v1","addr":"172.21.15.60:6819","nonce":2394404319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":2394404319},{"type":"v1","addr":"172.21.15.60:6823","nonce":2394404319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2394404319},{"type":"v1","addr":"172.21.15.60:6821","nonce":2394404319}]},"public_addr":"172.21.15.60:6817/2394404319","cluster_addr":"172.21.15.60:6819/2394404319","heartbeat_back_addr":"172.21.15.60:6823/2394404319","heartbeat_front_addr":"172.21.15.60:6821/2394404319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:59.583845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:59.538461+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:59.727579+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T23:49:12.397 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-21T23:49:12.397 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-21T23:49:12.398 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-21T23:49:12.559 INFO:teuthology.orchestra.run.smithi060.stdout:34359738373 2024-09-21T23:49:12.560 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-21T23:49:12.561 INFO:teuthology.orchestra.run.smithi060.stdout:34359738373 2024-09-21T23:49:12.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-21T23:49:12.562 INFO:teuthology.orchestra.run.smithi060.stdout:34359738374 2024-09-21T23:49:12.563 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-21T23:49:12.849 INFO:teuthology.orchestra.run.smithi060.stdout:34359738372 2024-09-21T23:49:12.872 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.1 2024-09-21T23:49:12.900 INFO:teuthology.orchestra.run.smithi060.stdout:34359738372 2024-09-21T23:49:12.909 INFO:teuthology.orchestra.run.smithi060.stdout:34359738373 2024-09-21T23:49:12.918 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2024-09-21T23:49:12.926 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2024-09-21T23:49:13.873 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-21T23:49:13.919 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-21T23:49:13.927 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-21T23:49:14.195 INFO:teuthology.orchestra.run.smithi060.stdout:34359738372 2024-09-21T23:49:14.213 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.1 2024-09-21T23:49:14.226 INFO:teuthology.orchestra.run.smithi060.stdout:34359738372 2024-09-21T23:49:14.243 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2024-09-21T23:49:14.266 INFO:teuthology.orchestra.run.smithi060.stdout:34359738373 2024-09-21T23:49:14.284 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2024-09-21T23:49:15.214 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-21T23:49:15.244 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-21T23:49:15.285 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-21T23:49:15.535 INFO:teuthology.orchestra.run.smithi060.stdout:34359738374 2024-09-21T23:49:15.554 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738374 for osd.1 2024-09-21T23:49:15.554 DEBUG:teuthology.parallel:result is None 2024-09-21T23:49:15.570 INFO:teuthology.orchestra.run.smithi060.stdout:34359738374 2024-09-21T23:49:15.589 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738374 for osd.2 2024-09-21T23:49:15.589 DEBUG:teuthology.parallel:result is None 2024-09-21T23:49:15.623 INFO:teuthology.orchestra.run.smithi060.stdout:34359738375 2024-09-21T23:49:15.641 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738375 for osd.0 2024-09-21T23:49:15.642 DEBUG:teuthology.parallel:result is None 2024-09-21T23:49:15.642 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-21T23:49:15.642 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-21T23:49:15.983 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:49:15.983 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-21T23:49:15.999 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-21T23:49:14.251681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95128,"kb_used_data":1552,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020392,"statfs":{"total":289910292480,"available":289812881408,"internally_reserved":0,"allocated":1589248,"data_stored":1295959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.893820"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:49:11.367983+0000","last_change":"2024-09-21T23:49:11.367983+0000","last_active":"2024-09-21T23:49:11.367983+0000","last_peered":"2024-09-21T23:49:11.367983+0000","last_clean":"2024-09-21T23:49:11.367983+0000","last_became_active":"2024-09-21T23:49:09.367547+0000","last_became_peered":"2024-09-21T23:49:09.367547+0000","last_unstale":"2024-09-21T23:49:11.367983+0000","last_undegraded":"2024-09-21T23:49:11.367983+0000","last_fullsized":"2024-09-21T23:49:11.367983+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:29:09.993126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00181346,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:49:11.367837+0000","last_change":"2024-09-21T23:49:11.367837+0000","last_active":"2024-09-21T23:49:11.367837+0000","last_peered":"2024-09-21T23:49:11.367837+0000","last_clean":"2024-09-21T23:49:11.367837+0000","last_became_active":"2024-09-21T23:49:09.371121+0000","last_became_peered":"2024-09-21T23:49:09.371121+0000","last_unstale":"2024-09-21T23:49:11.367837+0000","last_undegraded":"2024-09-21T23:49:11.367837+0000","last_fullsized":"2024-09-21T23:49:11.367837+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:20:14.639018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018013790000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:49:11.367216+0000","last_change":"2024-09-21T23:49:11.367216+0000","last_active":"2024-09-21T23:49:11.367216+0000","last_peered":"2024-09-21T23:49:11.367216+0000","last_clean":"2024-09-21T23:49:11.367216+0000","last_became_active":"2024-09-21T23:49:09.370827+0000","last_became_peered":"2024-09-21T23:49:09.370827+0000","last_unstale":"2024-09-21T23:49:11.367216+0000","last_undegraded":"2024-09-21T23:49:11.367216+0000","last_fullsized":"2024-09-21T23:49:11.367216+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:37:03.538645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011996000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:49:11.368537+0000","last_change":"2024-09-21T23:49:11.368537+0000","last_active":"2024-09-21T23:49:11.368537+0000","last_peered":"2024-09-21T23:49:11.368537+0000","last_clean":"2024-09-21T23:49:11.368537+0000","last_became_active":"2024-09-21T23:49:09.363532+0000","last_became_peered":"2024-09-21T23:49:09.363532+0000","last_unstale":"2024-09-21T23:49:11.368537+0000","last_undegraded":"2024-09-21T23:49:11.368537+0000","last_fullsized":"2024-09-21T23:49:11.368537+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T11:41:34.822852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022342730000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:49:11.378506+0000","last_change":"2024-09-21T23:49:11.376892+0000","last_active":"2024-09-21T23:49:11.378506+0000","last_peered":"2024-09-21T23:49:11.378506+0000","last_clean":"2024-09-21T23:49:11.378506+0000","last_became_active":"2024-09-21T23:49:09.365084+0000","last_became_peered":"2024-09-21T23:49:09.365084+0000","last_unstale":"2024-09-21T23:49:11.378506+0000","last_undegraded":"2024-09-21T23:49:11.378506+0000","last_fullsized":"2024-09-21T23:49:11.378506+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:38:10.347807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021899129999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:49:11.657951+0000","last_change":"2024-09-21T23:49:11.657951+0000","last_active":"2024-09-21T23:49:11.657951+0000","last_peered":"2024-09-21T23:49:11.657951+0000","last_clean":"2024-09-21T23:49:11.657951+0000","last_became_active":"2024-09-21T23:49:09.368367+0000","last_became_peered":"2024-09-21T23:49:09.368367+0000","last_unstale":"2024-09-21T23:49:11.657951+0000","last_undegraded":"2024-09-21T23:49:11.657951+0000","last_fullsized":"2024-09-21T23:49:11.657951+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:23:41.547546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091250999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:49:11.657611+0000","last_change":"2024-09-21T23:49:11.657611+0000","last_active":"2024-09-21T23:49:11.657611+0000","last_peered":"2024-09-21T23:49:11.657611+0000","last_clean":"2024-09-21T23:49:11.657611+0000","last_became_active":"2024-09-21T23:49:09.366135+0000","last_became_peered":"2024-09-21T23:49:09.366135+0000","last_unstale":"2024-09-21T23:49:11.657611+0000","last_undegraded":"2024-09-21T23:49:11.657611+0000","last_fullsized":"2024-09-21T23:49:11.657611+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:06:48.237231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074559400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:49:11.367169+0000","last_change":"2024-09-21T23:49:11.367169+0000","last_active":"2024-09-21T23:49:11.367169+0000","last_peered":"2024-09-21T23:49:11.367169+0000","last_clean":"2024-09-21T23:49:11.367169+0000","last_became_active":"2024-09-21T23:49:09.365442+0000","last_became_peered":"2024-09-21T23:49:09.365442+0000","last_unstale":"2024-09-21T23:49:11.367169+0000","last_undegraded":"2024-09-21T23:49:11.367169+0000","last_fullsized":"2024-09-21T23:49:11.367169+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:30:16.716709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001193475,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:49:11.365703+0000","last_change":"2024-09-21T23:49:05.633330+0000","last_active":"2024-09-21T23:49:11.365703+0000","last_peered":"2024-09-21T23:49:11.365703+0000","last_clean":"2024-09-21T23:49:11.365703+0000","last_became_active":"2024-09-21T23:49:05.630853+0000","last_became_peered":"2024-09-21T23:49:05.630853+0000","last_unstale":"2024-09-21T23:49:11.365703+0000","last_undegraded":"2024-09-21T23:49:11.365703+0000","last_fullsized":"2024-09-21T23:49:11.365703+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:04.328167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:04.328167+0000","last_clean_scrub_stamp":"2024-09-21T23:49:04.328167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:15:57.461205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31320,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340520,"statfs":{"total":96636764160,"available":96604692480,"internally_reserved":0,"allocated":131072,"data_stored":38395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:49:16.000 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-21T23:49:16.302 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:49:16.302 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-21T23:49:16.318 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-21T23:49:16.252197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95128,"kb_used_data":1552,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020392,"statfs":{"total":289910292480,"available":289812881408,"internally_reserved":0,"allocated":1589248,"data_stored":1295959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.897719"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:49:11.367983+0000","last_change":"2024-09-21T23:49:11.367983+0000","last_active":"2024-09-21T23:49:11.367983+0000","last_peered":"2024-09-21T23:49:11.367983+0000","last_clean":"2024-09-21T23:49:11.367983+0000","last_became_active":"2024-09-21T23:49:09.367547+0000","last_became_peered":"2024-09-21T23:49:09.367547+0000","last_unstale":"2024-09-21T23:49:11.367983+0000","last_undegraded":"2024-09-21T23:49:11.367983+0000","last_fullsized":"2024-09-21T23:49:11.367983+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:29:09.993126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00181346,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:49:11.367837+0000","last_change":"2024-09-21T23:49:11.367837+0000","last_active":"2024-09-21T23:49:11.367837+0000","last_peered":"2024-09-21T23:49:11.367837+0000","last_clean":"2024-09-21T23:49:11.367837+0000","last_became_active":"2024-09-21T23:49:09.371121+0000","last_became_peered":"2024-09-21T23:49:09.371121+0000","last_unstale":"2024-09-21T23:49:11.367837+0000","last_undegraded":"2024-09-21T23:49:11.367837+0000","last_fullsized":"2024-09-21T23:49:11.367837+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:20:14.639018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018013790000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:49:11.367216+0000","last_change":"2024-09-21T23:49:11.367216+0000","last_active":"2024-09-21T23:49:11.367216+0000","last_peered":"2024-09-21T23:49:11.367216+0000","last_clean":"2024-09-21T23:49:11.367216+0000","last_became_active":"2024-09-21T23:49:09.370827+0000","last_became_peered":"2024-09-21T23:49:09.370827+0000","last_unstale":"2024-09-21T23:49:11.367216+0000","last_undegraded":"2024-09-21T23:49:11.367216+0000","last_fullsized":"2024-09-21T23:49:11.367216+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:37:03.538645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011996000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:49:11.368537+0000","last_change":"2024-09-21T23:49:11.368537+0000","last_active":"2024-09-21T23:49:11.368537+0000","last_peered":"2024-09-21T23:49:11.368537+0000","last_clean":"2024-09-21T23:49:11.368537+0000","last_became_active":"2024-09-21T23:49:09.363532+0000","last_became_peered":"2024-09-21T23:49:09.363532+0000","last_unstale":"2024-09-21T23:49:11.368537+0000","last_undegraded":"2024-09-21T23:49:11.368537+0000","last_fullsized":"2024-09-21T23:49:11.368537+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T11:41:34.822852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022342730000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:49:11.378506+0000","last_change":"2024-09-21T23:49:11.376892+0000","last_active":"2024-09-21T23:49:11.378506+0000","last_peered":"2024-09-21T23:49:11.378506+0000","last_clean":"2024-09-21T23:49:11.378506+0000","last_became_active":"2024-09-21T23:49:09.365084+0000","last_became_peered":"2024-09-21T23:49:09.365084+0000","last_unstale":"2024-09-21T23:49:11.378506+0000","last_undegraded":"2024-09-21T23:49:11.378506+0000","last_fullsized":"2024-09-21T23:49:11.378506+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:38:10.347807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021899129999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:49:11.657951+0000","last_change":"2024-09-21T23:49:11.657951+0000","last_active":"2024-09-21T23:49:11.657951+0000","last_peered":"2024-09-21T23:49:11.657951+0000","last_clean":"2024-09-21T23:49:11.657951+0000","last_became_active":"2024-09-21T23:49:09.368367+0000","last_became_peered":"2024-09-21T23:49:09.368367+0000","last_unstale":"2024-09-21T23:49:11.657951+0000","last_undegraded":"2024-09-21T23:49:11.657951+0000","last_fullsized":"2024-09-21T23:49:11.657951+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:23:41.547546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091250999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:49:11.657611+0000","last_change":"2024-09-21T23:49:11.657611+0000","last_active":"2024-09-21T23:49:11.657611+0000","last_peered":"2024-09-21T23:49:11.657611+0000","last_clean":"2024-09-21T23:49:11.657611+0000","last_became_active":"2024-09-21T23:49:09.366135+0000","last_became_peered":"2024-09-21T23:49:09.366135+0000","last_unstale":"2024-09-21T23:49:11.657611+0000","last_undegraded":"2024-09-21T23:49:11.657611+0000","last_fullsized":"2024-09-21T23:49:11.657611+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:06:48.237231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074559400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:49:11.367169+0000","last_change":"2024-09-21T23:49:11.367169+0000","last_active":"2024-09-21T23:49:11.367169+0000","last_peered":"2024-09-21T23:49:11.367169+0000","last_clean":"2024-09-21T23:49:11.367169+0000","last_became_active":"2024-09-21T23:49:09.365442+0000","last_became_peered":"2024-09-21T23:49:09.365442+0000","last_unstale":"2024-09-21T23:49:11.367169+0000","last_undegraded":"2024-09-21T23:49:11.367169+0000","last_fullsized":"2024-09-21T23:49:11.367169+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:30:16.716709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001193475,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-21T23:49:11.365703+0000","last_change":"2024-09-21T23:49:05.633330+0000","last_active":"2024-09-21T23:49:11.365703+0000","last_peered":"2024-09-21T23:49:11.365703+0000","last_clean":"2024-09-21T23:49:11.365703+0000","last_became_active":"2024-09-21T23:49:05.630853+0000","last_became_peered":"2024-09-21T23:49:05.630853+0000","last_unstale":"2024-09-21T23:49:11.365703+0000","last_undegraded":"2024-09-21T23:49:11.365703+0000","last_fullsized":"2024-09-21T23:49:11.365703+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:04.328167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:04.328167+0000","last_clean_scrub_stamp":"2024-09-21T23:49:04.328167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:15:57.461205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31320,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340520,"statfs":{"total":96636764160,"available":96604692480,"internally_reserved":0,"allocated":131072,"data_stored":38395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T23:49:16.319 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-21T23:49:16.319 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-21T23:49:16.319 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-21T23:49:16.319 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-21T23:49:16.642 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-21T23:49:16.642 INFO:teuthology.orchestra.run.smithi060.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T23:49:16.660 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-21T23:49:16.660 INFO:teuthology.run_tasks:Running task exec... 2024-09-21T23:49:16.669 INFO:teuthology.task.exec:Executing custom commands... 2024-09-21T23:49:16.669 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:49:16.669 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-21T23:49:17.050 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.4] 2024-09-21T23:49:17.050 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAvW+9m5tRVHRAA1vmM21fxPjNZffmsHIplag== 2024-09-21T23:49:17.050 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-09-21T23:49:17.050 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-09-21T23:49:17.050 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.4 2024-09-21T23:49:17.070 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-21T23:49:17.452 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.5] 2024-09-21T23:49:17.452 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAvW+9mDcV7IhAACVd+QdvIxr23YBcsJEhV5A== 2024-09-21T23:49:17.452 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-09-21T23:49:17.453 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-09-21T23:49:17.453 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.5 2024-09-21T23:49:17.472 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-21T23:49:17.856 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.6] 2024-09-21T23:49:17.856 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAvW+9miQOrJxAAJQqmRh/nc09M9zj5phiZ6A== 2024-09-21T23:49:17.857 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-09-21T23:49:17.857 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-09-21T23:49:17.857 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.6 2024-09-21T23:49:17.876 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-21T23:49:18.252 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.4] 2024-09-21T23:49:18.252 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBXW+9mqQUmOBAAjhl1qSrOSF81zHMSpeCh+A== 2024-09-21T23:49:18.252 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-09-21T23:49:18.252 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-09-21T23:49:18.252 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.4 2024-09-21T23:49:18.270 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-21T23:49:18.636 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.5] 2024-09-21T23:49:18.636 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBYW+9mSoLVARAAbKB7H6nAkyK8VseLrgDXWA== 2024-09-21T23:49:18.636 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-09-21T23:49:18.636 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-09-21T23:49:18.637 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.5 2024-09-21T23:49:18.654 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-21T23:49:19.014 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.6] 2024-09-21T23:49:19.015 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBYW+9mmwRbBxAA0WPOQvz+nX3lPohhnABUyA== 2024-09-21T23:49:19.015 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-09-21T23:49:19.015 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-09-21T23:49:19.015 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.6 2024-09-21T23:49:19.032 INFO:teuthology.run_tasks:Running task exec... 2024-09-21T23:49:19.042 INFO:teuthology.task.exec:Executing custom commands... 2024-09-21T23:49:19.042 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi060.front.sepia.ceph.com 2024-09-21T23:49:19.042 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-21T23:49:19.407 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror] 2024-09-21T23:49:19.407 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAvW+9mVwhRAhAAv3lMBhBssBkuQcr7V3TJaA== 2024-09-21T23:49:19.407 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-21T23:49:19.407 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-09-21T23:49:19.408 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror 2024-09-21T23:49:19.427 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-21T23:49:19.789 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.0] 2024-09-21T23:49:19.789 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAvW+9m2h1RCBAAMAmYHS0D8IXT1ojXA+aT1g== 2024-09-21T23:49:19.790 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-09-21T23:49:19.790 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-09-21T23:49:19.790 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.0 2024-09-21T23:49:19.807 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-21T23:49:20.187 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.1] 2024-09-21T23:49:20.187 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAvW+9mOxC4DRAAYgGf6X6pKLUp9W16J8vU5w== 2024-09-21T23:49:20.187 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-09-21T23:49:20.187 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-09-21T23:49:20.187 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.1 2024-09-21T23:49:20.205 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-21T23:49:20.585 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.2] 2024-09-21T23:49:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAvW+9mZ1PuEhAAZ3GKlWlz79R0Xg/RzDafWA== 2024-09-21T23:49:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-09-21T23:49:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-09-21T23:49:20.586 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.2 2024-09-21T23:49:20.604 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-21T23:49:20.972 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.3] 2024-09-21T23:49:20.972 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAvW+9msPDxFxAARXF5nbGlg0aQKrR9i58lOw== 2024-09-21T23:49:20.972 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-09-21T23:49:20.972 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-09-21T23:49:20.972 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.3 2024-09-21T23:49:20.990 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-21T23:49:21.351 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror] 2024-09-21T23:49:21.351 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBXW+9m0FGLHhAAiYfDrXzwt0ukbgD5+GVVUA== 2024-09-21T23:49:21.351 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-21T23:49:21.351 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-09-21T23:49:21.351 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror 2024-09-21T23:49:21.369 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-21T23:49:21.740 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.0] 2024-09-21T23:49:21.740 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBXW+9mYuu4IxAA1iOGymSfzQHGEJNa49CcgQ== 2024-09-21T23:49:21.740 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-09-21T23:49:21.740 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-09-21T23:49:21.740 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.0 2024-09-21T23:49:21.757 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-21T23:49:22.121 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.1] 2024-09-21T23:49:22.121 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBXW+9mgAzZKBAAp+tDgHH4phx+4lMhvqe2Tw== 2024-09-21T23:49:22.121 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-09-21T23:49:22.121 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-09-21T23:49:22.121 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.1 2024-09-21T23:49:22.141 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-21T23:49:22.514 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.2] 2024-09-21T23:49:22.515 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBXW+9mp1PGLRAADnmAGQBmFmQ6acQYCdwerQ== 2024-09-21T23:49:22.515 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-09-21T23:49:22.515 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-09-21T23:49:22.515 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.2 2024-09-21T23:49:22.533 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-21T23:49:22.912 INFO:teuthology.orchestra.run.smithi060.stdout:[client.mirror.3] 2024-09-21T23:49:22.912 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBXW+9mc6YjMxAAW0aBNo5H6aXNuDuIBRXBfw== 2024-09-21T23:49:22.912 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "profile rbd-mirror" 2024-09-21T23:49:22.912 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "profile rbd" 2024-09-21T23:49:22.912 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.mirror.3 2024-09-21T23:49:22.932 INFO:teuthology.run_tasks:Running task workunit... 2024-09-21T23:49:22.950 INFO:tasks.workunit:Pulling workunits from ref 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T23:49:22.951 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-21T23:49:22.951 DEBUG:teuthology.orchestra.run.smithi060:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-21T23:49:22.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T23:49:22.957 INFO:teuthology.orchestra.run.smithi060.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-21T23:49:22.957 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-21T23:49:23.006 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-21T23:49:23.007 DEBUG:teuthology.orchestra.run.smithi060:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-21T23:49:23.055 INFO:tasks.workunit:timeout=3h 2024-09-21T23:49:23.055 INFO:tasks.workunit:cleanup=True 2024-09-21T23:49:23.055 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T23:49:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-21T23:50:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Updating files: 98% (12752/12896) Updating files: 99% (12768/12896) Updating files: 100% (12896/12896) Updating files: 100% (12896/12896), done. 2024-09-21T23:50:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Note: switching to '4e45e6b490e6cbe082cdf155fea780438ec53fc7'. 2024-09-21T23:50:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-09-21T23:50:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-21T23:50:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-21T23:50:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:state without impacting any branches by switching back to a branch. 2024-09-21T23:50:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-09-21T23:50:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-21T23:50:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-21T23:50:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-09-21T23:50:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: git switch -c 2024-09-21T23:50:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-09-21T23:50:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Or undo this operation with: 2024-09-21T23:50:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-09-21T23:50:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: git switch - 2024-09-21T23:50:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-09-21T23:50:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-21T23:50:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-09-21T23:50:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:HEAD is now at 4e45e6b490e Merge pull request #59907 from zdover23/wip-doc-2024-09-21-backport-59904-to-squid 2024-09-21T23:50:17.910 DEBUG:teuthology.orchestra.run.smithi060:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-21T23:50:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-21T23:50:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-21T23:50:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-21T23:50:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-21T23:50:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-21T23:50:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-21T23:50:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-21T23:50:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-21T23:50:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-21T23:50:18.942 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-21T23:50:18.942 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-21T23:50:18.989 INFO:tasks.workunit:Running workunits matching rbd/compare_mirror_images.sh on cluster1.client.mirror... 2024-09-21T23:50:18.990 INFO:tasks.workunit:Running workunit rbd/compare_mirror_images.sh... 2024-09-21T23:50:18.991 DEBUG:teuthology.orchestra.run.smithi060:workunit test rbd/compare_mirror_images.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e45e6b490e6cbe082cdf155fea780438ec53fc7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=krbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_images.sh 2024-09-21T23:50:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG_PREFIX=image-primary 2024-09-21T23:50:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MIRROR_IMAGE_MODE=snapshot 2024-09-21T23:50:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MIRROR_POOL_MODE=image 2024-09-21T23:50:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT_PREFIX=test-primary 2024-09-21T23:50:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2024-09-21T23:50:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ RBD_MIRROR_INSTANCES=1 2024-09-21T23:50:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ RBD_MIRROR_MODE=snapshot 2024-09-21T23:50:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ RBD_MIRROR_USE_EXISTING_CLUSTER=1 2024-09-21T23:50:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_images.sh 2024-09-21T23:50:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-21T23:50:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ type xmlstarlet 2024-09-21T23:50:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ XMLSTARLET=xmlstarlet 2024-09-21T23:50:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ RBD_MIRROR_INSTANCES=1 2024-09-21T23:50:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CLUSTER1=cluster1 2024-09-21T23:50:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CLUSTER2=cluster2 2024-09-21T23:50:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-21T23:50:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ POOL=mirror 2024-09-21T23:50:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ PARENT_POOL=mirror_parent 2024-09-21T23:50:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ NS1=ns1 2024-09-21T23:50:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ NS2=ns2 2024-09-21T23:50:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ TEMPDIR= 2024-09-21T23:50:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ID=mirror 2024-09-21T23:50:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2024-09-21T23:50:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-21T23:50:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ RBD_MIRROR_MODE=snapshot 2024-09-21T23:50:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ MIRROR_POOL_MODE=image 2024-09-21T23:50:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' snapshot = snapshot ']' 2024-09-21T23:50:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ MIRROR_POOL_MODE=image 2024-09-21T23:50:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-21T23:50:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS='--id mirror' 2024-09-21T23:50:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ LAST_MIRROR_INSTANCE=0 2024-09-21T23:50:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_images.sh 2024-09-21T23:50:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-21T23:50:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-21T23:50:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_BIN=. 2024-09-21T23:50:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_SRC=. 2024-09-21T23:50:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-21T23:50:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 -gt 0 ']' 2024-09-21T23:50:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ setup 2024-09-21T23:50:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local c 2024-09-21T23:50:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-21T23:50:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ setup_tempdir 2024-09-21T23:50:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:50:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ mktemp -d 2024-09-21T23:50:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ TEMPDIR=/tmp/tmp.5MnvHRcpWM 2024-09-21T23:50:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -z 1 ']' 2024-09-21T23:50:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ setup_pools cluster1 cluster2 2024-09-21T23:50:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local remote_cluster=cluster2 2024-09-21T23:50:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mon_map_file 2024-09-21T23:50:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mon_addr 2024-09-21T23:50:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local admin_key_file 2024-09-21T23:50:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local uuid 2024-09-21T23:50:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-09-21T23:50:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-21T23:50:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:pool 'mirror' already exists 2024-09-21T23:50:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-09-21T23:50:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-21T23:50:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:pool 'mirror_parent' already exists 2024-09-21T23:50:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-09-21T23:50:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-21T23:50:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-09-21T23:50:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-21T23:50:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:50:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-09-21T23:50:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-09-21T23:50:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-21T23:50:26.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-21T23:50:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-21T23:50:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-09-21T23:50:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-21T23:50:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -z ']' 2024-09-21T23:50:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -z ']' 2024-09-21T23:50:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ peer_add cluster1 mirror cluster2 2024-09-21T23:50:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local client_cluster=cluster2 2024-09-21T23:50:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local remote_cluster=cluster2 2024-09-21T23:50:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local uuid_var_name 2024-09-21T23:50:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:50:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local error_code 2024-09-21T23:50:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local peer_uuid 2024-09-21T23:50:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 16 32 2024-09-21T23:50:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-09-21T23:50:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-21T23:50:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ peer_uuid=cb35b424-608d-44ad-b1c4-1d406a5c5d08 2024-09-21T23:50:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=0 2024-09-21T23:50:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-09-21T23:50:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 0 -eq 17 ']' 2024-09-21T23:50:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 0 -eq 0 2024-09-21T23:50:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:50:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:50:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-21T23:50:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror_parent 2024-09-21T23:50:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local client_cluster=cluster2 2024-09-21T23:50:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local remote_cluster=cluster2 2024-09-21T23:50:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local uuid_var_name 2024-09-21T23:50:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:50:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local error_code 2024-09-21T23:50:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local peer_uuid 2024-09-21T23:50:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 16 32 2024-09-21T23:50:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-09-21T23:50:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-21T23:50:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ peer_uuid=48cc0769-9c4a-4926-b8e7-3f18e03f520f 2024-09-21T23:50:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=0 2024-09-21T23:50:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-09-21T23:50:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 0 -eq 17 ']' 2024-09-21T23:50:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 0 -eq 0 2024-09-21T23:50:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:50:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:50:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ setup_pools cluster2 cluster1 2024-09-21T23:50:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:50:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local remote_cluster=cluster1 2024-09-21T23:50:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mon_map_file 2024-09-21T23:50:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mon_addr 2024-09-21T23:50:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local admin_key_file 2024-09-21T23:50:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local uuid 2024-09-21T23:50:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-09-21T23:50:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-21T23:50:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:pool 'mirror' already exists 2024-09-21T23:50:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-09-21T23:50:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-21T23:50:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:pool 'mirror_parent' already exists 2024-09-21T23:50:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-09-21T23:50:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-21T23:50:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-09-21T23:50:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-21T23:50:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:50:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-09-21T23:50:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-09-21T23:50:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-21T23:50:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-21T23:50:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-21T23:50:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-09-21T23:50:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-21T23:50:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -z ']' 2024-09-21T23:50:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -z ']' 2024-09-21T23:50:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ peer_add cluster2 mirror cluster1 2024-09-21T23:50:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:50:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local client_cluster=cluster1 2024-09-21T23:50:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local remote_cluster=cluster1 2024-09-21T23:50:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local uuid_var_name 2024-09-21T23:50:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:50:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local error_code 2024-09-21T23:50:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local peer_uuid 2024-09-21T23:50:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 16 32 2024-09-21T23:50:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-09-21T23:50:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-21T23:50:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ peer_uuid=f33be45a-02fa-415d-8a13-89558963514a 2024-09-21T23:50:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=0 2024-09-21T23:50:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-09-21T23:50:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 0 -eq 17 ']' 2024-09-21T23:50:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 0 -eq 0 2024-09-21T23:50:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:50:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:50:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-21T23:50:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:50:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror_parent 2024-09-21T23:50:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local client_cluster=cluster1 2024-09-21T23:50:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local remote_cluster=cluster1 2024-09-21T23:50:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local uuid_var_name 2024-09-21T23:50:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:50:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local error_code 2024-09-21T23:50:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local peer_uuid 2024-09-21T23:50:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 16 32 2024-09-21T23:50:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-09-21T23:50:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-21T23:50:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ peer_uuid=b6703952-dcd8-4297-8085-4d609cdda46b 2024-09-21T23:50:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ error_code=0 2024-09-21T23:50:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set -e 2024-09-21T23:50:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 0 -eq 17 ']' 2024-09-21T23:50:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test 0 -eq 0 2024-09-21T23:50:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:50:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:50:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:50:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ start_mirrors cluster1 2024-09-21T23:50:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 0 0 2024-09-21T23:50:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-21T23:50:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ start_mirror cluster1:0 2024-09-21T23:50:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1:0 2024-09-21T23:50:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance 2024-09-21T23:50:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-21T23:50:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local val=cluster1:0 2024-09-21T23:50:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster_var_name=cluster 2024-09-21T23:50:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance_var_name=instance 2024-09-21T23:50:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cluster=cluster1 2024-09-21T23:50:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ instance=0 2024-09-21T23:50:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-21T23:50:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval cluster=cluster1 2024-09-21T23:50:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-09-21T23:50:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval instance=0 2024-09-21T23:50:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-09-21T23:50:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n '' 2024-09-21T23:50:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-21T23:50:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ start_mirrors cluster2 2024-09-21T23:50:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:50:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 0 0 2024-09-21T23:50:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-21T23:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ start_mirror cluster2:0 2024-09-21T23:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2:0 2024-09-21T23:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance 2024-09-21T23:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-21T23:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local val=cluster2:0 2024-09-21T23:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster_var_name=cluster 2024-09-21T23:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local instance_var_name=instance 2024-09-21T23:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cluster=cluster2 2024-09-21T23:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ instance=0 2024-09-21T23:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-21T23:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval cluster=cluster2 2024-09-21T23:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster2 2024-09-21T23:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ eval instance=0 2024-09-21T23:50:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-09-21T23:50:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n '' 2024-09-21T23:50:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-21T23:50:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wget https://download.ceph.com/qa/linux-5.4.tar.gz 2024-09-21T23:50:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:--2024-09-21 23:50:35-- https://download.ceph.com/qa/linux-5.4.tar.gz 2024-09-21T23:50:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-09-21T23:50:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:443... connected. 2024-09-21T23:50:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:HTTP request sent, awaiting response... 200 OK 2024-09-21T23:50:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Length: 172616875 (165M) [application/octet-stream] 2024-09-21T23:50:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:Saving to: 'linux-5.4.tar.gz' 2024-09-21T23:50:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-09-21T23:50:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 0K .......... .......... .......... .......... .......... 0% 2.38M 69s 2024-09-21T23:50:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50K .......... .......... .......... .......... .......... 0% 2.39M 69s 2024-09-21T23:50:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 100K .......... .......... .......... .......... .......... 0% 182M 46s 2024-09-21T23:50:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 150K .......... .......... .......... .......... .......... 0% 2.41M 52s 2024-09-21T23:50:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 200K .......... .......... .......... .......... .......... 0% 225M 42s 2024-09-21T23:50:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 250K .......... .......... .......... .......... .......... 0% 281M 35s 2024-09-21T23:50:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 300K .......... .......... .......... .......... .......... 0% 2.46M 39s 2024-09-21T23:50:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 350K .......... .......... .......... .......... .......... 0% 145M 34s 2024-09-21T23:50:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 400K .......... .......... .......... .......... .......... 0% 354M 31s 2024-09-21T23:50:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 450K .......... .......... .......... .......... .......... 0% 354M 28s 2024-09-21T23:50:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 500K .......... .......... .......... .......... .......... 0% 384M 25s 2024-09-21T23:50:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 550K .......... .......... .......... .......... .......... 0% 311M 23s 2024-09-21T23:50:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 600K .......... .......... .......... .......... .......... 0% 399M 21s 2024-09-21T23:50:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 650K .......... .......... .......... .......... .......... 0% 55.9M 20s 2024-09-21T23:50:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 700K .......... .......... .......... .......... .......... 0% 2.61M 23s 2024-09-21T23:50:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 750K .......... .......... .......... .......... .......... 0% 214M 22s 2024-09-21T23:50:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 800K .......... .......... .......... .......... .......... 0% 258M 20s 2024-09-21T23:50:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 850K .......... .......... .......... .......... .......... 0% 285M 19s 2024-09-21T23:50:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 900K .......... .......... .......... .......... .......... 0% 351M 18s 2024-09-21T23:50:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 950K .......... .......... .......... .......... .......... 0% 355M 17s 2024-09-21T23:50:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1000K .......... .......... .......... .......... .......... 0% 54.4M 17s 2024-09-21T23:50:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1050K .......... .......... .......... .......... .......... 0% 2.64M 19s 2024-09-21T23:50:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1100K .......... .......... .......... .......... .......... 0% 158M 18s 2024-09-21T23:50:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1150K .......... .......... .......... .......... .......... 0% 146M 17s 2024-09-21T23:50:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1200K .......... .......... .......... .......... .......... 0% 181M 17s 2024-09-21T23:50:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1250K .......... .......... .......... .......... .......... 0% 228M 16s 2024-09-21T23:50:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1300K .......... .......... .......... .......... .......... 0% 247M 15s 2024-09-21T23:50:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1350K .......... .......... .......... .......... .......... 0% 225M 15s 2024-09-21T23:50:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1400K .......... .......... .......... .......... .......... 0% 53.4M 14s 2024-09-21T23:50:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1450K .......... .......... .......... .......... .......... 0% 31.8M 14s 2024-09-21T23:50:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1500K .......... .......... .......... .......... .......... 0% 2.89M 15s 2024-09-21T23:50:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1550K .......... .......... .......... .......... .......... 0% 74.1M 15s 2024-09-21T23:50:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1600K .......... .......... .......... .......... .......... 0% 146M 15s 2024-09-21T23:50:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1650K .......... .......... .......... .......... .......... 1% 129M 14s 2024-09-21T23:50:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1700K .......... .......... .......... .......... .......... 1% 151M 14s 2024-09-21T23:50:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1750K .......... .......... .......... .......... .......... 1% 117M 13s 2024-09-21T23:50:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1800K .......... .......... .......... .......... .......... 1% 147M 13s 2024-09-21T23:50:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1850K .......... .......... .......... .......... .......... 1% 140M 13s 2024-09-21T23:50:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1900K .......... .......... .......... .......... .......... 1% 31.3M 13s 2024-09-21T23:50:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 1950K .......... .......... .......... .......... .......... 1% 3.03M 14s 2024-09-21T23:50:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2000K .......... .......... .......... .......... .......... 1% 129M 13s 2024-09-21T23:50:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2050K .......... .......... .......... .......... .......... 1% 126M 13s 2024-09-21T23:50:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2100K .......... .......... .......... .......... .......... 1% 151M 13s 2024-09-21T23:50:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2150K .......... .......... .......... .......... .......... 1% 138M 13s 2024-09-21T23:50:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2200K .......... .......... .......... .......... .......... 1% 154M 12s 2024-09-21T23:50:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2250K .......... .......... .......... .......... .......... 1% 154M 12s 2024-09-21T23:50:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2300K .......... .......... .......... .......... .......... 1% 40.8M 12s 2024-09-21T23:50:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2350K .......... .......... .......... .......... .......... 1% 24.6M 12s 2024-09-21T23:50:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2400K .......... .......... .......... .......... .......... 1% 3.23M 12s 2024-09-21T23:50:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2450K .......... .......... .......... .......... .......... 1% 117M 12s 2024-09-21T23:50:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2500K .......... .......... .......... .......... .......... 1% 136M 12s 2024-09-21T23:50:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2550K .......... .......... .......... .......... .......... 1% 131M 12s 2024-09-21T23:50:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2600K .......... .......... .......... .......... .......... 1% 149M 12s 2024-09-21T23:50:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2650K .......... .......... .......... .......... .......... 1% 154M 11s 2024-09-21T23:50:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2700K .......... .......... .......... .......... .......... 1% 153M 11s 2024-09-21T23:50:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2750K .......... .......... .......... .......... .......... 1% 42.3M 11s 2024-09-21T23:50:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2800K .......... .......... .......... .......... .......... 1% 38.1M 11s 2024-09-21T23:50:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2850K .......... .......... .......... .......... .......... 1% 33.3M 11s 2024-09-21T23:50:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2900K .......... .......... .......... .......... .......... 1% 3.36M 11s 2024-09-21T23:50:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2950K .......... .......... .......... .......... .......... 1% 109M 11s 2024-09-21T23:50:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3000K .......... .......... .......... .......... .......... 1% 155M 11s 2024-09-21T23:50:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3050K .......... .......... .......... .......... .......... 1% 163M 11s 2024-09-21T23:50:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3100K .......... .......... .......... .......... .......... 1% 166M 11s 2024-09-21T23:50:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3150K .......... .......... .......... .......... .......... 1% 93.0M 11s 2024-09-21T23:50:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3200K .......... .......... .......... .......... .......... 1% 101M 11s 2024-09-21T23:50:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3250K .......... .......... .......... .......... .......... 1% 59.1M 10s 2024-09-21T23:50:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3300K .......... .......... .......... .......... .......... 1% 29.9M 10s 2024-09-21T23:50:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3350K .......... .......... .......... .......... .......... 2% 24.4M 10s 2024-09-21T23:50:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3400K .......... .......... .......... .......... .......... 2% 3.59M 11s 2024-09-21T23:50:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3450K .......... .......... .......... .......... .......... 2% 76.3M 11s 2024-09-21T23:50:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3500K .......... .......... .......... .......... .......... 2% 80.2M 11s 2024-09-21T23:50:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3550K .......... .......... .......... .......... .......... 2% 69.7M 10s 2024-09-21T23:50:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3600K .......... .......... .......... .......... .......... 2% 81.2M 10s 2024-09-21T23:50:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3650K .......... .......... .......... .......... .......... 2% 81.5M 10s 2024-09-21T23:50:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3700K .......... .......... .......... .......... .......... 2% 78.5M 10s 2024-09-21T23:50:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3750K .......... .......... .......... .......... .......... 2% 70.4M 10s 2024-09-21T23:50:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3800K .......... .......... .......... .......... .......... 2% 85.3M 10s 2024-09-21T23:50:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3850K .......... .......... .......... .......... .......... 2% 34.5M 10s 2024-09-21T23:50:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3900K .......... .......... .......... .......... .......... 2% 40.8M 10s 2024-09-21T23:50:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 3950K .......... .......... .......... .......... .......... 2% 3.75M 10s 2024-09-21T23:50:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4000K .......... .......... .......... .......... .......... 2% 79.9M 10s 2024-09-21T23:50:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4050K .......... .......... .......... .......... .......... 2% 80.3M 10s 2024-09-21T23:50:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4100K .......... .......... .......... .......... .......... 2% 81.5M 10s 2024-09-21T23:50:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4150K .......... .......... .......... .......... .......... 2% 73.6M 10s 2024-09-21T23:50:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4200K .......... .......... .......... .......... .......... 2% 65.9M 10s 2024-09-21T23:50:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4250K .......... .......... .......... .......... .......... 2% 81.8M 10s 2024-09-21T23:50:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4300K .......... .......... .......... .......... .......... 2% 84.2M 9s 2024-09-21T23:50:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4350K .......... .......... .......... .......... .......... 2% 69.1M 9s 2024-09-21T23:50:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4400K .......... .......... .......... .......... .......... 2% 46.9M 9s 2024-09-21T23:50:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4450K .......... .......... .......... .......... .......... 2% 28.5M 9s 2024-09-21T23:50:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4500K .......... .......... .......... .......... .......... 2% 3.92M 10s 2024-09-21T23:50:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4550K .......... .......... .......... .......... .......... 2% 71.4M 10s 2024-09-21T23:50:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4600K .......... .......... .......... .......... .......... 2% 80.4M 9s 2024-09-21T23:50:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4650K .......... .......... .......... .......... .......... 2% 81.2M 9s 2024-09-21T23:50:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4700K .......... .......... .......... .......... .......... 2% 83.0M 9s 2024-09-21T23:50:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4750K .......... .......... .......... .......... .......... 2% 72.2M 9s 2024-09-21T23:50:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4800K .......... .......... .......... .......... .......... 2% 83.1M 9s 2024-09-21T23:50:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4850K .......... .......... .......... .......... .......... 2% 83.9M 9s 2024-09-21T23:50:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4900K .......... .......... .......... .......... .......... 2% 80.6M 9s 2024-09-21T23:50:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 4950K .......... .......... .......... .......... .......... 2% 46.2M 9s 2024-09-21T23:50:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5000K .......... .......... .......... .......... .......... 2% 26.1M 9s 2024-09-21T23:50:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5050K .......... .......... .......... .......... .......... 3% 37.6M 9s 2024-09-21T23:50:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5100K .......... .......... .......... .......... .......... 3% 4.33M 9s 2024-09-21T23:50:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5150K .......... .......... .......... .......... .......... 3% 61.6M 9s 2024-09-21T23:50:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5200K .......... .......... .......... .......... .......... 3% 80.8M 9s 2024-09-21T23:50:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5250K .......... .......... .......... .......... .......... 3% 78.6M 9s 2024-09-21T23:50:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5300K .......... .......... .......... .......... .......... 3% 84.8M 9s 2024-09-21T23:50:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5350K .......... .......... .......... .......... .......... 3% 73.0M 9s 2024-09-21T23:50:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5400K .......... .......... .......... .......... .......... 3% 85.1M 9s 2024-09-21T23:50:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5450K .......... .......... .......... .......... .......... 3% 83.1M 9s 2024-09-21T23:50:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5500K .......... .......... .......... .......... .......... 3% 84.2M 9s 2024-09-21T23:50:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5550K .......... .......... .......... .......... .......... 3% 27.2M 9s 2024-09-21T23:50:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5600K .......... .......... .......... .......... .......... 3% 36.6M 8s 2024-09-21T23:50:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5650K .......... .......... .......... .......... .......... 3% 26.0M 8s 2024-09-21T23:50:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5700K .......... .......... .......... .......... .......... 3% 4.83M 9s 2024-09-21T23:50:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5750K .......... .......... .......... .......... .......... 3% 36.0M 9s 2024-09-21T23:50:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5800K .......... .......... .......... .......... .......... 3% 76.8M 9s 2024-09-21T23:50:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5850K .......... .......... .......... .......... .......... 3% 80.7M 9s 2024-09-21T23:50:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5900K .......... .......... .......... .......... .......... 3% 85.0M 8s 2024-09-21T23:50:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 5950K .......... .......... .......... .......... .......... 3% 68.8M 8s 2024-09-21T23:50:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6000K .......... .......... .......... .......... .......... 3% 81.0M 8s 2024-09-21T23:50:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6050K .......... .......... .......... .......... .......... 3% 82.6M 8s 2024-09-21T23:50:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6100K .......... .......... .......... .......... .......... 3% 82.3M 8s 2024-09-21T23:50:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6150K .......... .......... .......... .......... .......... 3% 44.5M 8s 2024-09-21T23:50:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6200K .......... .......... .......... .......... .......... 3% 46.6M 8s 2024-09-21T23:50:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6250K .......... .......... .......... .......... .......... 3% 25.3M 8s 2024-09-21T23:50:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6300K .......... .......... .......... .......... .......... 3% 32.8M 8s 2024-09-21T23:50:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6350K .......... .......... .......... .......... .......... 3% 5.30M 8s 2024-09-21T23:50:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6400K .......... .......... .......... .......... .......... 3% 34.4M 8s 2024-09-21T23:50:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6450K .......... .......... .......... .......... .......... 3% 74.3M 8s 2024-09-21T23:50:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6500K .......... .......... .......... .......... .......... 3% 85.0M 8s 2024-09-21T23:50:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6550K .......... .......... .......... .......... .......... 3% 72.7M 8s 2024-09-21T23:50:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6600K .......... .......... .......... .......... .......... 3% 82.3M 8s 2024-09-21T23:50:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6650K .......... .......... .......... .......... .......... 3% 84.4M 8s 2024-09-21T23:50:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6700K .......... .......... .......... .......... .......... 4% 86.3M 8s 2024-09-21T23:50:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6750K .......... .......... .......... .......... .......... 4% 49.4M 8s 2024-09-21T23:50:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6800K .......... .......... .......... .......... .......... 4% 24.1M 8s 2024-09-21T23:50:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6850K .......... .......... .......... .......... .......... 4% 32.1M 8s 2024-09-21T23:50:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6900K .......... .......... .......... .......... .......... 4% 26.1M 8s 2024-09-21T23:50:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 6950K .......... .......... .......... .......... .......... 4% 51.3M 8s 2024-09-21T23:50:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7000K .......... .......... .......... .......... .......... 4% 25.0M 8s 2024-09-21T23:50:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7050K .......... .......... .......... .......... .......... 4% 7.61M 8s 2024-09-21T23:50:35.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7100K .......... .......... .......... .......... .......... 4% 48.8M 8s 2024-09-21T23:50:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7150K .......... .......... .......... .......... .......... 4% 69.1M 8s 2024-09-21T23:50:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7200K .......... .......... .......... .......... .......... 4% 81.2M 8s 2024-09-21T23:50:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7250K .......... .......... .......... .......... .......... 4% 85.0M 8s 2024-09-21T23:50:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7300K .......... .......... .......... .......... .......... 4% 47.8M 8s 2024-09-21T23:50:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7350K .......... .......... .......... .......... .......... 4% 22.6M 8s 2024-09-21T23:50:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7400K .......... .......... .......... .......... .......... 4% 39.4M 8s 2024-09-21T23:50:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7450K .......... .......... .......... .......... .......... 4% 43.4M 8s 2024-09-21T23:50:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7500K .......... .......... .......... .......... .......... 4% 24.4M 8s 2024-09-21T23:50:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7550K .......... .......... .......... .......... .......... 4% 25.1M 8s 2024-09-21T23:50:35.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7600K .......... .......... .......... .......... .......... 4% 27.2M 8s 2024-09-21T23:50:35.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7650K .......... .......... .......... .......... .......... 4% 28.0M 8s 2024-09-21T23:50:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7700K .......... .......... .......... .......... .......... 4% 12.1M 8s 2024-09-21T23:50:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7750K .......... .......... .......... .......... .......... 4% 69.4M 8s 2024-09-21T23:50:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7800K .......... .......... .......... .......... .......... 4% 77.1M 8s 2024-09-21T23:50:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7850K .......... .......... .......... .......... .......... 4% 48.0M 8s 2024-09-21T23:50:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7900K .......... .......... .......... .......... .......... 4% 24.3M 8s 2024-09-21T23:50:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 7950K .......... .......... .......... .......... .......... 4% 24.1M 8s 2024-09-21T23:50:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8000K .......... .......... .......... .......... .......... 4% 57.9M 7s 2024-09-21T23:50:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8050K .......... .......... .......... .......... .......... 4% 25.6M 7s 2024-09-21T23:50:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8100K .......... .......... .......... .......... .......... 4% 31.1M 7s 2024-09-21T23:50:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8150K .......... .......... .......... .......... .......... 4% 22.6M 7s 2024-09-21T23:50:35.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8200K .......... .......... .......... .......... .......... 4% 27.0M 7s 2024-09-21T23:50:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8250K .......... .......... .......... .......... .......... 4% 57.0M 7s 2024-09-21T23:50:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8300K .......... .......... .......... .......... .......... 4% 27.7M 7s 2024-09-21T23:50:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8350K .......... .......... .......... .......... .......... 4% 20.5M 7s 2024-09-21T23:50:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8400K .......... .......... .......... .......... .......... 5% 35.0M 7s 2024-09-21T23:50:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8450K .......... .......... .......... .......... .......... 5% 22.5M 7s 2024-09-21T23:50:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8500K .......... .......... .......... .......... .......... 5% 83.6M 7s 2024-09-21T23:50:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8550K .......... .......... .......... .......... .......... 5% 22.5M 7s 2024-09-21T23:50:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8600K .......... .......... .......... .......... .......... 5% 32.2M 7s 2024-09-21T23:50:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8650K .......... .......... .......... .......... .......... 5% 24.3M 7s 2024-09-21T23:50:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8700K .......... .......... .......... .......... .......... 5% 24.7M 7s 2024-09-21T23:50:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8750K .......... .......... .......... .......... .......... 5% 23.6M 7s 2024-09-21T23:50:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8800K .......... .......... .......... .......... .......... 5% 69.9M 7s 2024-09-21T23:50:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8850K .......... .......... .......... .......... .......... 5% 31.2M 7s 2024-09-21T23:50:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8900K .......... .......... .......... .......... .......... 5% 24.1M 7s 2024-09-21T23:50:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 8950K .......... .......... .......... .......... .......... 5% 25.0M 7s 2024-09-21T23:50:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9000K .......... .......... .......... .......... .......... 5% 54.3M 7s 2024-09-21T23:50:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9050K .......... .......... .......... .......... .......... 5% 24.7M 7s 2024-09-21T23:50:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9100K .......... .......... .......... .......... .......... 5% 32.0M 7s 2024-09-21T23:50:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9150K .......... .......... .......... .......... .......... 5% 25.9M 7s 2024-09-21T23:50:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9200K .......... .......... .......... .......... .......... 5% 24.5M 7s 2024-09-21T23:50:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9250K .......... .......... .......... .......... .......... 5% 23.2M 7s 2024-09-21T23:50:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9300K .......... .......... .......... .......... .......... 5% 60.0M 7s 2024-09-21T23:50:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9350K .......... .......... .......... .......... .......... 5% 30.6M 7s 2024-09-21T23:50:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9400K .......... .......... .......... .......... .......... 5% 9.17M 7s 2024-09-21T23:50:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9450K .......... .......... .......... .......... .......... 5% 98.8M 7s 2024-09-21T23:50:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9500K .......... .......... .......... .......... .......... 5% 117M 7s 2024-09-21T23:50:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9550K .......... .......... .......... .......... .......... 5% 74.8M 7s 2024-09-21T23:50:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9600K .......... .......... .......... .......... .......... 5% 24.7M 7s 2024-09-21T23:50:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9650K .......... .......... .......... .......... .......... 5% 34.5M 7s 2024-09-21T23:50:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9700K .......... .......... .......... .......... .......... 5% 29.6M 7s 2024-09-21T23:50:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9750K .......... .......... .......... .......... .......... 5% 34.9M 7s 2024-09-21T23:50:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9800K .......... .......... .......... .......... .......... 5% 26.6M 7s 2024-09-21T23:50:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9850K .......... .......... .......... .......... .......... 5% 37.0M 7s 2024-09-21T23:50:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9900K .......... .......... .......... .......... .......... 5% 22.0M 7s 2024-09-21T23:50:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 9950K .......... .......... .......... .......... .......... 5% 24.7M 7s 2024-09-21T23:50:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10000K .......... .......... .......... .......... .......... 5% 25.2M 7s 2024-09-21T23:50:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10050K .......... .......... .......... .......... .......... 5% 51.9M 7s 2024-09-21T23:50:35.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10100K .......... .......... .......... .......... .......... 6% 23.7M 7s 2024-09-21T23:50:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10150K .......... .......... .......... .......... .......... 6% 34.3M 7s 2024-09-21T23:50:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10200K .......... .......... .......... .......... .......... 6% 29.3M 7s 2024-09-21T23:50:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10250K .......... .......... .......... .......... .......... 6% 38.7M 7s 2024-09-21T23:50:35.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10300K .......... .......... .......... .......... .......... 6% 25.2M 7s 2024-09-21T23:50:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10350K .......... .......... .......... .......... .......... 6% 24.1M 7s 2024-09-21T23:50:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10400K .......... .......... .......... .......... .......... 6% 36.6M 7s 2024-09-21T23:50:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10450K .......... .......... .......... .......... .......... 6% 22.5M 7s 2024-09-21T23:50:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10500K .......... .......... .......... .......... .......... 6% 24.8M 7s 2024-09-21T23:50:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10550K .......... .......... .......... .......... .......... 6% 59.1M 7s 2024-09-21T23:50:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10600K .......... .......... .......... .......... .......... 6% 22.6M 7s 2024-09-21T23:50:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10650K .......... .......... .......... .......... .......... 6% 35.5M 7s 2024-09-21T23:50:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10700K .......... .......... .......... .......... .......... 6% 24.5M 7s 2024-09-21T23:50:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10750K .......... .......... .......... .......... .......... 6% 24.1M 7s 2024-09-21T23:50:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10800K .......... .......... .......... .......... .......... 6% 51.3M 7s 2024-09-21T23:50:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10850K .......... .......... .......... .......... .......... 6% 24.9M 7s 2024-09-21T23:50:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10900K .......... .......... .......... .......... .......... 6% 33.3M 7s 2024-09-21T23:50:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 10950K .......... .......... .......... .......... .......... 6% 25.9M 7s 2024-09-21T23:50:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11000K .......... .......... .......... .......... .......... 6% 28.9M 7s 2024-09-21T23:50:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11050K .......... .......... .......... .......... .......... 6% 35.8M 7s 2024-09-21T23:50:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11100K .......... .......... .......... .......... .......... 6% 35.8M 7s 2024-09-21T23:50:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11150K .......... .......... .......... .......... .......... 6% 20.1M 7s 2024-09-21T23:50:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11200K .......... .......... .......... .......... .......... 6% 29.1M 7s 2024-09-21T23:50:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11250K .......... .......... .......... .......... .......... 6% 25.3M 7s 2024-09-21T23:50:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11300K .......... .......... .......... .......... .......... 6% 53.0M 7s 2024-09-21T23:50:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11350K .......... .......... .......... .......... .......... 6% 24.0M 7s 2024-09-21T23:50:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11400K .......... .......... .......... .......... .......... 6% 34.1M 7s 2024-09-21T23:50:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11450K .......... .......... .......... .......... .......... 6% 23.6M 7s 2024-09-21T23:50:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11500K .......... .......... .......... .......... .......... 6% 68.8M 7s 2024-09-21T23:50:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11550K .......... .......... .......... .......... .......... 6% 15.0M 7s 2024-09-21T23:50:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11600K .......... .......... .......... .......... .......... 6% 60.0M 7s 2024-09-21T23:50:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11650K .......... .......... .......... .......... .......... 6% 32.2M 7s 2024-09-21T23:50:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11700K .......... .......... .......... .......... .......... 6% 24.4M 7s 2024-09-21T23:50:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11750K .......... .......... .......... .......... .......... 7% 23.7M 7s 2024-09-21T23:50:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11800K .......... .......... .......... .......... .......... 7% 54.6M 7s 2024-09-21T23:50:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11850K .......... .......... .......... .......... .......... 7% 24.9M 7s 2024-09-21T23:50:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11900K .......... .......... .......... .......... .......... 7% 36.3M 7s 2024-09-21T23:50:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 11950K .......... .......... .......... .......... .......... 7% 24.0M 7s 2024-09-21T23:50:35.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12000K .......... .......... .......... .......... .......... 7% 23.8M 7s 2024-09-21T23:50:35.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12050K .......... .......... .......... .......... .......... 7% 53.1M 7s 2024-09-21T23:50:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12100K .......... .......... .......... .......... .......... 7% 24.9M 7s 2024-09-21T23:50:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12150K .......... .......... .......... .......... .......... 7% 32.6M 7s 2024-09-21T23:50:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12200K .......... .......... .......... .......... .......... 7% 24.7M 7s 2024-09-21T23:50:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12250K .......... .......... .......... .......... .......... 7% 23.9M 7s 2024-09-21T23:50:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12300K .......... .......... .......... .......... .......... 7% 53.2M 7s 2024-09-21T23:50:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12350K .......... .......... .......... .......... .......... 7% 23.7M 7s 2024-09-21T23:50:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12400K .......... .......... .......... .......... .......... 7% 25.5M 7s 2024-09-21T23:50:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12450K .......... .......... .......... .......... .......... 7% 37.7M 7s 2024-09-21T23:50:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12500K .......... .......... .......... .......... .......... 7% 28.8M 7s 2024-09-21T23:50:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12550K .......... .......... .......... .......... .......... 7% 32.0M 7s 2024-09-21T23:50:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12600K .......... .......... .......... .......... .......... 7% 25.4M 7s 2024-09-21T23:50:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12650K .......... .......... .......... .......... .......... 7% 23.9M 7s 2024-09-21T23:50:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12700K .......... .......... .......... .......... .......... 7% 35.1M 7s 2024-09-21T23:50:35.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12750K .......... .......... .......... .......... .......... 7% 24.0M 7s 2024-09-21T23:50:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12800K .......... .......... .......... .......... .......... 7% 25.1M 7s 2024-09-21T23:50:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12850K .......... .......... .......... .......... .......... 7% 60.8M 6s 2024-09-21T23:50:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12900K .......... .......... .......... .......... .......... 7% 23.1M 6s 2024-09-21T23:50:35.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 12950K .......... .......... .......... .......... .......... 7% 33.8M 6s 2024-09-21T23:50:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13000K .......... .......... .......... .......... .......... 7% 25.2M 6s 2024-09-21T23:50:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13050K .......... .......... .......... .......... .......... 7% 49.0M 6s 2024-09-21T23:50:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13100K .......... .......... .......... .......... .......... 7% 25.2M 6s 2024-09-21T23:50:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13150K .......... .......... .......... .......... .......... 7% 34.2M 6s 2024-09-21T23:50:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13200K .......... .......... .......... .......... .......... 7% 24.8M 6s 2024-09-21T23:50:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13250K .......... .......... .......... .......... .......... 7% 23.7M 6s 2024-09-21T23:50:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13300K .......... .......... .......... .......... .......... 7% 25.0M 6s 2024-09-21T23:50:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13350K .......... .......... .......... .......... .......... 7% 51.0M 6s 2024-09-21T23:50:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13400K .......... .......... .......... .......... .......... 7% 34.9M 6s 2024-09-21T23:50:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13450K .......... .......... .......... .......... .......... 8% 23.3M 6s 2024-09-21T23:50:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13500K .......... .......... .......... .......... .......... 8% 28.9M 6s 2024-09-21T23:50:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13550K .......... .......... .......... .......... .......... 8% 21.5M 6s 2024-09-21T23:50:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13600K .......... .......... .......... .......... .......... 8% 49.6M 6s 2024-09-21T23:50:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13650K .......... .......... .......... .......... .......... 8% 25.1M 6s 2024-09-21T23:50:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13700K .......... .......... .......... .......... .......... 8% 36.3M 6s 2024-09-21T23:50:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13750K .......... .......... .......... .......... .......... 8% 24.2M 6s 2024-09-21T23:50:36.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13800K .......... .......... .......... .......... .......... 8% 23.2M 6s 2024-09-21T23:50:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13850K .......... .......... .......... .......... .......... 8% 58.1M 6s 2024-09-21T23:50:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13900K .......... .......... .......... .......... .......... 8% 33.8M 6s 2024-09-21T23:50:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 13950K .......... .......... .......... .......... .......... 8% 23.1M 6s 2024-09-21T23:50:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14000K .......... .......... .......... .......... .......... 8% 24.5M 6s 2024-09-21T23:50:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14050K .......... .......... .......... .......... .......... 8% 27.1M 6s 2024-09-21T23:50:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14100K .......... .......... .......... .......... .......... 8% 42.8M 6s 2024-09-21T23:50:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14150K .......... .......... .......... .......... .......... 8% 24.2M 6s 2024-09-21T23:50:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14200K .......... .......... .......... .......... .......... 8% 36.9M 6s 2024-09-21T23:50:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14250K .......... .......... .......... .......... .......... 8% 24.1M 6s 2024-09-21T23:50:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14300K .......... .......... .......... .......... .......... 8% 50.7M 6s 2024-09-21T23:50:36.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14350K .......... .......... .......... .......... .......... 8% 15.6M 6s 2024-09-21T23:50:36.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14400K .......... .......... .......... .......... .......... 8% 82.3M 6s 2024-09-21T23:50:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14450K .......... .......... .......... .......... .......... 8% 27.6M 6s 2024-09-21T23:50:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14500K .......... .......... .......... .......... .......... 8% 23.8M 6s 2024-09-21T23:50:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14550K .......... .......... .......... .......... .......... 8% 24.8M 6s 2024-09-21T23:50:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14600K .......... .......... .......... .......... .......... 8% 53.7M 6s 2024-09-21T23:50:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14650K .......... .......... .......... .......... .......... 8% 23.9M 6s 2024-09-21T23:50:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14700K .......... .......... .......... .......... .......... 8% 36.5M 6s 2024-09-21T23:50:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14750K .......... .......... .......... .......... .......... 8% 24.0M 6s 2024-09-21T23:50:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14800K .......... .......... .......... .......... .......... 8% 24.0M 6s 2024-09-21T23:50:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14850K .......... .......... .......... .......... .......... 8% 50.7M 6s 2024-09-21T23:50:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14900K .......... .......... .......... .......... .......... 8% 25.8M 6s 2024-09-21T23:50:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 14950K .......... .......... .......... .......... .......... 8% 21.6M 6s 2024-09-21T23:50:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15000K .......... .......... .......... .......... .......... 8% 31.2M 6s 2024-09-21T23:50:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15050K .......... .......... .......... .......... .......... 8% 50.3M 6s 2024-09-21T23:50:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15100K .......... .......... .......... .......... .......... 8% 31.7M 6s 2024-09-21T23:50:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15150K .......... .......... .......... .......... .......... 9% 22.5M 6s 2024-09-21T23:50:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15200K .......... .......... .......... .......... .......... 9% 25.2M 6s 2024-09-21T23:50:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15250K .......... .......... .......... .......... .......... 9% 36.1M 6s 2024-09-21T23:50:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15300K .......... .......... .......... .......... .......... 9% 23.9M 6s 2024-09-21T23:50:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15350K .......... .......... .......... .......... .......... 9% 47.1M 6s 2024-09-21T23:50:36.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15400K .......... .......... .......... .......... .......... 9% 23.0M 6s 2024-09-21T23:50:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15450K .......... .......... .......... .......... .......... 9% 43.5M 6s 2024-09-21T23:50:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15500K .......... .......... .......... .......... .......... 9% 22.2M 6s 2024-09-21T23:50:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15550K .......... .......... .......... .......... .......... 9% 23.5M 6s 2024-09-21T23:50:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15600K .......... .......... .......... .......... .......... 9% 26.3M 6s 2024-09-21T23:50:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15650K .......... .......... .......... .......... .......... 9% 57.9M 6s 2024-09-21T23:50:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15700K .......... .......... .......... .......... .......... 9% 22.8M 6s 2024-09-21T23:50:36.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15750K .......... .......... .......... .......... .......... 9% 33.4M 6s 2024-09-21T23:50:36.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15800K .......... .......... .......... .......... .......... 9% 24.5M 6s 2024-09-21T23:50:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15850K .......... .......... .......... .......... .......... 9% 53.2M 6s 2024-09-21T23:50:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15900K .......... .......... .......... .......... .......... 9% 24.7M 6s 2024-09-21T23:50:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 15950K .......... .......... .......... .......... .......... 9% 22.3M 6s 2024-09-21T23:50:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16000K .......... .......... .......... .......... .......... 9% 37.8M 6s 2024-09-21T23:50:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16050K .......... .......... .......... .......... .......... 9% 24.4M 6s 2024-09-21T23:50:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16100K .......... .......... .......... .......... .......... 9% 24.4M 6s 2024-09-21T23:50:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16150K .......... .......... .......... .......... .......... 9% 49.0M 6s 2024-09-21T23:50:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16200K .......... .......... .......... .......... .......... 9% 32.6M 6s 2024-09-21T23:50:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16250K .......... .......... .......... .......... .......... 9% 24.3M 6s 2024-09-21T23:50:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16300K .......... .......... .......... .......... .......... 9% 43.8M 6s 2024-09-21T23:50:36.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16350K .......... .......... .......... .......... .......... 9% 22.0M 6s 2024-09-21T23:50:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16400K .......... .......... .......... .......... .......... 9% 33.3M 6s 2024-09-21T23:50:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16450K .......... .......... .......... .......... .......... 9% 21.9M 6s 2024-09-21T23:50:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16500K .......... .......... .......... .......... .......... 9% 39.3M 6s 2024-09-21T23:50:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16550K .......... .......... .......... .......... .......... 9% 25.1M 6s 2024-09-21T23:50:36.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16600K .......... .......... .......... .......... .......... 9% 48.3M 6s 2024-09-21T23:50:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16650K .......... .......... .......... .......... .......... 9% 23.7M 6s 2024-09-21T23:50:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16700K .......... .......... .......... .......... .......... 9% 35.5M 6s 2024-09-21T23:50:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16750K .......... .......... .......... .......... .......... 9% 23.3M 6s 2024-09-21T23:50:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16800K .......... .......... .......... .......... .......... 9% 25.5M 6s 2024-09-21T23:50:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16850K .......... .......... .......... .......... .......... 10% 24.8M 6s 2024-09-21T23:50:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16900K .......... .......... .......... .......... .......... 10% 49.7M 6s 2024-09-21T23:50:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 16950K .......... .......... .......... .......... .......... 10% 36.9M 6s 2024-09-21T23:50:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17000K .......... .......... .......... .......... .......... 10% 24.0M 6s 2024-09-21T23:50:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17050K .......... .......... .......... .......... .......... 10% 24.4M 6s 2024-09-21T23:50:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17100K .......... .......... .......... .......... .......... 10% 54.2M 6s 2024-09-21T23:50:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17150K .......... .......... .......... .......... .......... 10% 23.8M 6s 2024-09-21T23:50:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17200K .......... .......... .......... .......... .......... 10% 24.8M 6s 2024-09-21T23:50:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17250K .......... .......... .......... .......... .......... 10% 30.9M 6s 2024-09-21T23:50:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17300K .......... .......... .......... .......... .......... 10% 24.8M 6s 2024-09-21T23:50:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17350K .......... .......... .......... .......... .......... 10% 25.7M 6s 2024-09-21T23:50:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17400K .......... .......... .......... .......... .......... 10% 45.3M 6s 2024-09-21T23:50:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17450K .......... .......... .......... .......... .......... 10% 25.1M 6s 2024-09-21T23:50:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17500K .......... .......... .......... .......... .......... 10% 44.8M 6s 2024-09-21T23:50:36.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17550K .......... .......... .......... .......... .......... 10% 21.0M 6s 2024-09-21T23:50:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17600K .......... .......... .......... .......... .......... 10% 25.2M 6s 2024-09-21T23:50:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17650K .......... .......... .......... .......... .......... 10% 49.9M 6s 2024-09-21T23:50:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17700K .......... .......... .......... .......... .......... 10% 24.0M 6s 2024-09-21T23:50:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17750K .......... .......... .......... .......... .......... 10% 35.1M 6s 2024-09-21T23:50:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17800K .......... .......... .......... .......... .......... 10% 23.6M 6s 2024-09-21T23:50:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17850K .......... .......... .......... .......... .......... 10% 24.2M 6s 2024-09-21T23:50:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17900K .......... .......... .......... .......... .......... 10% 62.1M 6s 2024-09-21T23:50:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 17950K .......... .......... .......... .......... .......... 10% 22.3M 6s 2024-09-21T23:50:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18000K .......... .......... .......... .......... .......... 10% 24.6M 6s 2024-09-21T23:50:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18050K .......... .......... .......... .......... .......... 10% 36.7M 6s 2024-09-21T23:50:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18100K .......... .......... .......... .......... .......... 10% 24.6M 6s 2024-09-21T23:50:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18150K .......... .......... .......... .......... .......... 10% 46.0M 6s 2024-09-21T23:50:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18200K .......... .......... .......... .......... .......... 10% 25.7M 6s 2024-09-21T23:50:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18250K .......... .......... .......... .......... .......... 10% 36.2M 6s 2024-09-21T23:50:36.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18300K .......... .......... .......... .......... .......... 10% 23.6M 6s 2024-09-21T23:50:36.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18350K .......... .......... .......... .......... .......... 10% 23.1M 6s 2024-09-21T23:50:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18400K .......... .......... .......... .......... .......... 10% 25.5M 6s 2024-09-21T23:50:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18450K .......... .......... .......... .......... .......... 10% 57.6M 6s 2024-09-21T23:50:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18500K .......... .......... .......... .......... .......... 11% 32.3M 6s 2024-09-21T23:50:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18550K .......... .......... .......... .......... .......... 11% 24.2M 6s 2024-09-21T23:50:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18600K .......... .......... .......... .......... .......... 11% 23.5M 6s 2024-09-21T23:50:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18650K .......... .......... .......... .......... .......... 11% 54.8M 6s 2024-09-21T23:50:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18700K .......... .......... .......... .......... .......... 11% 24.1M 6s 2024-09-21T23:50:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18750K .......... .......... .......... .......... .......... 11% 24.6M 6s 2024-09-21T23:50:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18800K .......... .......... .......... .......... .......... 11% 34.5M 6s 2024-09-21T23:50:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18850K .......... .......... .......... .......... .......... 11% 34.9M 6s 2024-09-21T23:50:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18900K .......... .......... .......... .......... .......... 11% 18.5M 6s 2024-09-21T23:50:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 18950K .......... .......... .......... .......... .......... 11% 48.9M 6s 2024-09-21T23:50:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19000K .......... .......... .......... .......... .......... 11% 33.6M 6s 2024-09-21T23:50:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19050K .......... .......... .......... .......... .......... 11% 26.1M 6s 2024-09-21T23:50:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19100K .......... .......... .......... .......... .......... 11% 25.0M 6s 2024-09-21T23:50:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19150K .......... .......... .......... .......... .......... 11% 23.1M 6s 2024-09-21T23:50:36.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19200K .......... .......... .......... .......... .......... 11% 50.5M 6s 2024-09-21T23:50:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19250K .......... .......... .......... .......... .......... 11% 25.2M 6s 2024-09-21T23:50:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19300K .......... .......... .......... .......... .......... 11% 34.9M 6s 2024-09-21T23:50:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19350K .......... .......... .......... .......... .......... 11% 29.0M 6s 2024-09-21T23:50:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19400K .......... .......... .......... .......... .......... 11% 36.5M 6s 2024-09-21T23:50:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19450K .......... .......... .......... .......... .......... 11% 25.7M 6s 2024-09-21T23:50:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19500K .......... .......... .......... .......... .......... 11% 32.0M 6s 2024-09-21T23:50:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19550K .......... .......... .......... .......... .......... 11% 23.4M 6s 2024-09-21T23:50:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19600K .......... .......... .......... .......... .......... 11% 23.9M 6s 2024-09-21T23:50:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19650K .......... .......... .......... .......... .......... 11% 26.1M 6s 2024-09-21T23:50:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19700K .......... .......... .......... .......... .......... 11% 49.2M 6s 2024-09-21T23:50:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19750K .......... .......... .......... .......... .......... 11% 24.7M 6s 2024-09-21T23:50:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19800K .......... .......... .......... .......... .......... 11% 35.0M 6s 2024-09-21T23:50:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19850K .......... .......... .......... .......... .......... 11% 25.3M 6s 2024-09-21T23:50:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19900K .......... .......... .......... .......... .......... 11% 48.7M 6s 2024-09-21T23:50:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 19950K .......... .......... .......... .......... .......... 11% 23.6M 6s 2024-09-21T23:50:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20000K .......... .......... .......... .......... .......... 11% 23.9M 6s 2024-09-21T23:50:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20050K .......... .......... .......... .......... .......... 11% 34.1M 6s 2024-09-21T23:50:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20100K .......... .......... .......... .......... .......... 11% 24.5M 6s 2024-09-21T23:50:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20150K .......... .......... .......... .......... .......... 11% 25.0M 6s 2024-09-21T23:50:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20200K .......... .......... .......... .......... .......... 12% 51.5M 6s 2024-09-21T23:50:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20250K .......... .......... .......... .......... .......... 12% 25.0M 6s 2024-09-21T23:50:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20300K .......... .......... .......... .......... .......... 12% 34.1M 6s 2024-09-21T23:50:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20350K .......... .......... .......... .......... .......... 12% 18.2M 6s 2024-09-21T23:50:36.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20400K .......... .......... .......... .......... .......... 12% 38.5M 6s 2024-09-21T23:50:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20450K .......... .......... .......... .......... .......... 12% 49.2M 6s 2024-09-21T23:50:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20500K .......... .......... .......... .......... .......... 12% 24.7M 6s 2024-09-21T23:50:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20550K .......... .......... .......... .......... .......... 12% 35.5M 6s 2024-09-21T23:50:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20600K .......... .......... .......... .......... .......... 12% 22.7M 6s 2024-09-21T23:50:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20650K .......... .......... .......... .......... .......... 12% 25.1M 6s 2024-09-21T23:50:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20700K .......... .......... .......... .......... .......... 12% 53.0M 6s 2024-09-21T23:50:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20750K .......... .......... .......... .......... .......... 12% 24.9M 6s 2024-09-21T23:50:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20800K .......... .......... .......... .......... .......... 12% 22.9M 6s 2024-09-21T23:50:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20850K .......... .......... .......... .......... .......... 12% 35.9M 6s 2024-09-21T23:50:36.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20900K .......... .......... .......... .......... .......... 12% 25.1M 6s 2024-09-21T23:50:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 20950K .......... .......... .......... .......... .......... 12% 47.7M 6s 2024-09-21T23:50:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21000K .......... .......... .......... .......... .......... 12% 25.7M 6s 2024-09-21T23:50:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21050K .......... .......... .......... .......... .......... 12% 34.4M 6s 2024-09-21T23:50:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21100K .......... .......... .......... .......... .......... 12% 25.8M 6s 2024-09-21T23:50:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21150K .......... .......... .......... .......... .......... 12% 21.9M 6s 2024-09-21T23:50:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21200K .......... .......... .......... .......... .......... 12% 25.0M 6s 2024-09-21T23:50:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21250K .......... .......... .......... .......... .......... 12% 53.2M 6s 2024-09-21T23:50:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21300K .......... .......... .......... .......... .......... 12% 30.8M 6s 2024-09-21T23:50:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21350K .......... .......... .......... .......... .......... 12% 24.1M 6s 2024-09-21T23:50:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21400K .......... .......... .......... .......... .......... 12% 24.5M 6s 2024-09-21T23:50:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21450K .......... .......... .......... .......... .......... 12% 50.5M 6s 2024-09-21T23:50:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21500K .......... .......... .......... .......... .......... 12% 25.3M 6s 2024-09-21T23:50:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21550K .......... .......... .......... .......... .......... 12% 25.0M 6s 2024-09-21T23:50:36.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21600K .......... .......... .......... .......... .......... 12% 32.9M 6s 2024-09-21T23:50:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21650K .......... .......... .......... .......... .......... 12% 25.8M 6s 2024-09-21T23:50:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21700K .......... .......... .......... .......... .......... 12% 52.6M 6s 2024-09-21T23:50:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21750K .......... .......... .......... .......... .......... 12% 24.6M 6s 2024-09-21T23:50:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21800K .......... .......... .......... .......... .......... 12% 32.0M 6s 2024-09-21T23:50:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21850K .......... .......... .......... .......... .......... 12% 23.9M 6s 2024-09-21T23:50:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21900K .......... .......... .......... .......... .......... 13% 30.3M 6s 2024-09-21T23:50:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 21950K .......... .......... .......... .......... .......... 13% 20.2M 6s 2024-09-21T23:50:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22000K .......... .......... .......... .......... .......... 13% 50.0M 6s 2024-09-21T23:50:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22050K .......... .......... .......... .......... .......... 13% 24.8M 6s 2024-09-21T23:50:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22100K .......... .......... .......... .......... .......... 13% 35.7M 6s 2024-09-21T23:50:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22150K .......... .......... .......... .......... .......... 13% 24.0M 6s 2024-09-21T23:50:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22200K .......... .......... .......... .......... .......... 13% 50.4M 6s 2024-09-21T23:50:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22250K .......... .......... .......... .......... .......... 13% 23.3M 6s 2024-09-21T23:50:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22300K .......... .......... .......... .......... .......... 13% 40.1M 6s 2024-09-21T23:50:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22350K .......... .......... .......... .......... .......... 13% 22.4M 6s 2024-09-21T23:50:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22400K .......... .......... .......... .......... .......... 13% 25.6M 6s 2024-09-21T23:50:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22450K .......... .......... .......... .......... .......... 13% 24.2M 6s 2024-09-21T23:50:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22500K .......... .......... .......... .......... .......... 13% 50.0M 6s 2024-09-21T23:50:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22550K .......... .......... .......... .......... .......... 13% 25.4M 6s 2024-09-21T23:50:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22600K .......... .......... .......... .......... .......... 13% 34.5M 6s 2024-09-21T23:50:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22650K .......... .......... .......... .......... .......... 13% 24.5M 6s 2024-09-21T23:50:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22700K .......... .......... .......... .......... .......... 13% 50.8M 6s 2024-09-21T23:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22750K .......... .......... .......... .......... .......... 13% 23.7M 6s 2024-09-21T23:50:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22800K .......... .......... .......... .......... .......... 13% 24.6M 6s 2024-09-21T23:50:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22850K .......... .......... .......... .......... .......... 13% 35.8M 6s 2024-09-21T23:50:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22900K .......... .......... .......... .......... .......... 13% 22.9M 6s 2024-09-21T23:50:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 22950K .......... .......... .......... .......... .......... 13% 25.9M 6s 2024-09-21T23:50:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23000K .......... .......... .......... .......... .......... 13% 57.1M 6s 2024-09-21T23:50:36.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23050K .......... .......... .......... .......... .......... 13% 22.6M 6s 2024-09-21T23:50:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23100K .......... .......... .......... .......... .......... 13% 35.2M 6s 2024-09-21T23:50:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23150K .......... .......... .......... .......... .......... 13% 24.7M 6s 2024-09-21T23:50:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23200K .......... .......... .......... .......... .......... 13% 24.1M 6s 2024-09-21T23:50:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23250K .......... .......... .......... .......... .......... 13% 51.4M 6s 2024-09-21T23:50:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23300K .......... .......... .......... .......... .......... 13% 24.8M 6s 2024-09-21T23:50:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23350K .......... .......... .......... .......... .......... 13% 34.2M 6s 2024-09-21T23:50:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23400K .......... .......... .......... .......... .......... 13% 24.5M 6s 2024-09-21T23:50:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23450K .......... .......... .......... .......... .......... 13% 23.3M 6s 2024-09-21T23:50:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23500K .......... .......... .......... .......... .......... 13% 58.1M 6s 2024-09-21T23:50:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23550K .......... .......... .......... .......... .......... 14% 24.1M 6s 2024-09-21T23:50:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23600K .......... .......... .......... .......... .......... 14% 23.5M 6s 2024-09-21T23:50:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23650K .......... .......... .......... .......... .......... 14% 35.1M 6s 2024-09-21T23:50:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23700K .......... .......... .......... .......... .......... 14% 25.2M 6s 2024-09-21T23:50:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23750K .......... .......... .......... .......... .......... 14% 44.0M 6s 2024-09-21T23:50:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23800K .......... .......... .......... .......... .......... 14% 25.6M 6s 2024-09-21T23:50:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23850K .......... .......... .......... .......... .......... 14% 35.3M 5s 2024-09-21T23:50:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23900K .......... .......... .......... .......... .......... 14% 24.9M 5s 2024-09-21T23:50:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 23950K .......... .......... .......... .......... .......... 14% 24.2M 5s 2024-09-21T23:50:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24000K .......... .......... .......... .......... .......... 14% 23.0M 5s 2024-09-21T23:50:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24050K .......... .......... .......... .......... .......... 14% 61.1M 5s 2024-09-21T23:50:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24100K .......... .......... .......... .......... .......... 14% 25.5M 5s 2024-09-21T23:50:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24150K .......... .......... .......... .......... .......... 14% 29.8M 5s 2024-09-21T23:50:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24200K .......... .......... .......... .......... .......... 14% 24.8M 5s 2024-09-21T23:50:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24250K .......... .......... .......... .......... .......... 14% 55.0M 5s 2024-09-21T23:50:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24300K .......... .......... .......... .......... .......... 14% 25.1M 5s 2024-09-21T23:50:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24350K .......... .......... .......... .......... .......... 14% 23.0M 5s 2024-09-21T23:50:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24400K .......... .......... .......... .......... .......... 14% 35.4M 5s 2024-09-21T23:50:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24450K .......... .......... .......... .......... .......... 14% 27.2M 5s 2024-09-21T23:50:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24500K .......... .......... .......... .......... .......... 14% 45.2M 5s 2024-09-21T23:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24550K .......... .......... .......... .......... .......... 14% 23.3M 5s 2024-09-21T23:50:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24600K .......... .......... .......... .......... .......... 14% 35.2M 5s 2024-09-21T23:50:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24650K .......... .......... .......... .......... .......... 14% 23.7M 5s 2024-09-21T23:50:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24700K .......... .......... .......... .......... .......... 14% 24.2M 5s 2024-09-21T23:50:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24750K .......... .......... .......... .......... .......... 14% 24.7M 5s 2024-09-21T23:50:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24800K .......... .......... .......... .......... .......... 14% 48.6M 5s 2024-09-21T23:50:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24850K .......... .......... .......... .......... .......... 14% 25.3M 5s 2024-09-21T23:50:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24900K .......... .......... .......... .......... .......... 14% 34.6M 5s 2024-09-21T23:50:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 24950K .......... .......... .......... .......... .......... 14% 23.8M 5s 2024-09-21T23:50:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25000K .......... .......... .......... .......... .......... 14% 51.0M 5s 2024-09-21T23:50:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25050K .......... .......... .......... .......... .......... 14% 24.3M 5s 2024-09-21T23:50:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25100K .......... .......... .......... .......... .......... 14% 37.0M 5s 2024-09-21T23:50:36.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25150K .......... .......... .......... .......... .......... 14% 21.9M 5s 2024-09-21T23:50:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25200K .......... .......... .......... .......... .......... 14% 24.7M 5s 2024-09-21T23:50:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25250K .......... .......... .......... .......... .......... 15% 24.6M 5s 2024-09-21T23:50:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25300K .......... .......... .......... .......... .......... 15% 62.2M 5s 2024-09-21T23:50:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25350K .......... .......... .......... .......... .......... 15% 22.6M 5s 2024-09-21T23:50:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25400K .......... .......... .......... .......... .......... 15% 30.4M 5s 2024-09-21T23:50:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25450K .......... .......... .......... .......... .......... 15% 28.7M 5s 2024-09-21T23:50:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25500K .......... .......... .......... .......... .......... 15% 48.8M 5s 2024-09-21T23:50:36.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25550K .......... .......... .......... .......... .......... 15% 23.9M 5s 2024-09-21T23:50:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25600K .......... .......... .......... .......... .......... 15% 25.0M 5s 2024-09-21T23:50:36.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25650K .......... .......... .......... .......... .......... 15% 36.6M 5s 2024-09-21T23:50:36.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25700K .......... .......... .......... .......... .......... 15% 23.2M 5s 2024-09-21T23:50:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25750K .......... .......... .......... .......... .......... 15% 24.7M 5s 2024-09-21T23:50:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25800K .......... .......... .......... .......... .......... 15% 53.9M 5s 2024-09-21T23:50:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25850K .......... .......... .......... .......... .......... 15% 33.5M 5s 2024-09-21T23:50:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25900K .......... .......... .......... .......... .......... 15% 24.9M 5s 2024-09-21T23:50:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 25950K .......... .......... .......... .......... .......... 15% 24.2M 5s 2024-09-21T23:50:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26000K .......... .......... .......... .......... .......... 15% 24.1M 5s 2024-09-21T23:50:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26050K .......... .......... .......... .......... .......... 15% 50.2M 5s 2024-09-21T23:50:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26100K .......... .......... .......... .......... .......... 15% 24.9M 5s 2024-09-21T23:50:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26150K .......... .......... .......... .......... .......... 15% 33.7M 5s 2024-09-21T23:50:36.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26200K .......... .......... .......... .......... .......... 15% 24.8M 5s 2024-09-21T23:50:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26250K .......... .......... .......... .......... .......... 15% 23.0M 5s 2024-09-21T23:50:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26300K .......... .......... .......... .......... .......... 15% 62.7M 5s 2024-09-21T23:50:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26350K .......... .......... .......... .......... .......... 15% 24.0M 5s 2024-09-21T23:50:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26400K .......... .......... .......... .......... .......... 15% 35.2M 5s 2024-09-21T23:50:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26450K .......... .......... .......... .......... .......... 15% 23.3M 5s 2024-09-21T23:50:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26500K .......... .......... .......... .......... .......... 15% 25.2M 5s 2024-09-21T23:50:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26550K .......... .......... .......... .......... .......... 15% 48.6M 5s 2024-09-21T23:50:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26600K .......... .......... .......... .......... .......... 15% 25.3M 5s 2024-09-21T23:50:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26650K .......... .......... .......... .......... .......... 15% 38.7M 5s 2024-09-21T23:50:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26700K .......... .......... .......... .......... .......... 15% 22.8M 5s 2024-09-21T23:50:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26750K .......... .......... .......... .......... .......... 15% 24.1M 5s 2024-09-21T23:50:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26800K .......... .......... .......... .......... .......... 15% 23.2M 5s 2024-09-21T23:50:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26850K .......... .......... .......... .......... .......... 15% 56.7M 5s 2024-09-21T23:50:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26900K .......... .......... .......... .......... .......... 15% 26.2M 5s 2024-09-21T23:50:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 26950K .......... .......... .......... .......... .......... 16% 29.7M 5s 2024-09-21T23:50:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27000K .......... .......... .......... .......... .......... 16% 21.3M 5s 2024-09-21T23:50:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27050K .......... .......... .......... .......... .......... 16% 51.2M 5s 2024-09-21T23:50:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27100K .......... .......... .......... .......... .......... 16% 24.9M 5s 2024-09-21T23:50:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27150K .......... .......... .......... .......... .......... 16% 25.0M 5s 2024-09-21T23:50:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27200K .......... .......... .......... .......... .......... 16% 33.3M 5s 2024-09-21T23:50:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27250K .......... .......... .......... .......... .......... 16% 25.2M 5s 2024-09-21T23:50:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27300K .......... .......... .......... .......... .......... 16% 50.4M 5s 2024-09-21T23:50:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27350K .......... .......... .......... .......... .......... 16% 24.2M 5s 2024-09-21T23:50:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27400K .......... .......... .......... .......... .......... 16% 35.4M 5s 2024-09-21T23:50:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27450K .......... .......... .......... .......... .......... 16% 22.4M 5s 2024-09-21T23:50:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27500K .......... .......... .......... .......... .......... 16% 24.7M 5s 2024-09-21T23:50:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27550K .......... .......... .......... .......... .......... 16% 24.2M 5s 2024-09-21T23:50:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27600K .......... .......... .......... .......... .......... 16% 49.3M 5s 2024-09-21T23:50:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27650K .......... .......... .......... .......... .......... 16% 25.2M 5s 2024-09-21T23:50:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27700K .......... .......... .......... .......... .......... 16% 35.1M 5s 2024-09-21T23:50:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27750K .......... .......... .......... .......... .......... 16% 24.1M 5s 2024-09-21T23:50:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27800K .......... .......... .......... .......... .......... 16% 52.2M 5s 2024-09-21T23:50:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27850K .......... .......... .......... .......... .......... 16% 23.9M 5s 2024-09-21T23:50:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27900K .......... .......... .......... .......... .......... 16% 36.6M 5s 2024-09-21T23:50:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 27950K .......... .......... .......... .......... .......... 16% 24.2M 5s 2024-09-21T23:50:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28000K .......... .......... .......... .......... .......... 16% 22.9M 5s 2024-09-21T23:50:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28050K .......... .......... .......... .......... .......... 16% 49.0M 5s 2024-09-21T23:50:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28100K .......... .......... .......... .......... .......... 16% 28.9M 5s 2024-09-21T23:50:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28150K .......... .......... .......... .......... .......... 16% 21.6M 5s 2024-09-21T23:50:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28200K .......... .......... .......... .......... .......... 16% 35.8M 5s 2024-09-21T23:50:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28250K .......... .......... .......... .......... .......... 16% 25.8M 5s 2024-09-21T23:50:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28300K .......... .......... .......... .......... .......... 16% 51.4M 5s 2024-09-21T23:50:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28350K .......... .......... .......... .......... .......... 16% 23.2M 5s 2024-09-21T23:50:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28400K .......... .......... .......... .......... .......... 16% 23.9M 5s 2024-09-21T23:50:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28450K .......... .......... .......... .......... .......... 16% 38.4M 5s 2024-09-21T23:50:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28500K .......... .......... .......... .......... .......... 16% 25.2M 5s 2024-09-21T23:50:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28550K .......... .......... .......... .......... .......... 16% 22.6M 5s 2024-09-21T23:50:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28600K .......... .......... .......... .......... .......... 16% 55.5M 5s 2024-09-21T23:50:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28650K .......... .......... .......... .......... .......... 17% 33.3M 5s 2024-09-21T23:50:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28700K .......... .......... .......... .......... .......... 17% 24.8M 5s 2024-09-21T23:50:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28750K .......... .......... .......... .......... .......... 17% 22.9M 5s 2024-09-21T23:50:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28800K .......... .......... .......... .......... .......... 17% 26.4M 5s 2024-09-21T23:50:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28850K .......... .......... .......... .......... .......... 17% 58.1M 5s 2024-09-21T23:50:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28900K .......... .......... .......... .......... .......... 17% 23.0M 5s 2024-09-21T23:50:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 28950K .......... .......... .......... .......... .......... 17% 33.9M 5s 2024-09-21T23:50:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29000K .......... .......... .......... .......... .......... 17% 23.1M 5s 2024-09-21T23:50:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29050K .......... .......... .......... .......... .......... 17% 45.0M 5s 2024-09-21T23:50:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29100K .......... .......... .......... .......... .......... 17% 24.3M 5s 2024-09-21T23:50:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29150K .......... .......... .......... .......... .......... 17% 24.3M 5s 2024-09-21T23:50:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29200K .......... .......... .......... .......... .......... 17% 31.8M 5s 2024-09-21T23:50:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29250K .......... .......... .......... .......... .......... 17% 26.5M 5s 2024-09-21T23:50:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29300K .......... .......... .......... .......... .......... 17% 24.0M 5s 2024-09-21T23:50:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29350K .......... .......... .......... .......... .......... 17% 46.5M 5s 2024-09-21T23:50:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29400K .......... .......... .......... .......... .......... 17% 23.8M 5s 2024-09-21T23:50:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29450K .......... .......... .......... .......... .......... 17% 35.8M 5s 2024-09-21T23:50:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29500K .......... .......... .......... .......... .......... 17% 25.1M 5s 2024-09-21T23:50:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29550K .......... .......... .......... .......... .......... 17% 22.8M 5s 2024-09-21T23:50:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29600K .......... .......... .......... .......... .......... 17% 53.8M 5s 2024-09-21T23:50:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29650K .......... .......... .......... .......... .......... 17% 25.4M 5s 2024-09-21T23:50:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29700K .......... .......... .......... .......... .......... 17% 31.8M 5s 2024-09-21T23:50:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29750K .......... .......... .......... .......... .......... 17% 24.7M 5s 2024-09-21T23:50:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29800K .......... .......... .......... .......... .......... 17% 24.5M 5s 2024-09-21T23:50:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29850K .......... .......... .......... .......... .......... 17% 57.3M 5s 2024-09-21T23:50:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29900K .......... .......... .......... .......... .......... 17% 23.4M 5s 2024-09-21T23:50:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 29950K .......... .......... .......... .......... .......... 17% 24.4M 5s 2024-09-21T23:50:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30000K .......... .......... .......... .......... .......... 17% 35.1M 5s 2024-09-21T23:50:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30050K .......... .......... .......... .......... .......... 17% 24.6M 5s 2024-09-21T23:50:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30100K .......... .......... .......... .......... .......... 17% 50.0M 5s 2024-09-21T23:50:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30150K .......... .......... .......... .......... .......... 17% 24.4M 5s 2024-09-21T23:50:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30200K .......... .......... .......... .......... .......... 17% 27.1M 5s 2024-09-21T23:50:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30250K .......... .......... .......... .......... .......... 17% 31.0M 5s 2024-09-21T23:50:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30300K .......... .......... .......... .......... .......... 18% 23.9M 5s 2024-09-21T23:50:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30350K .......... .......... .......... .......... .......... 18% 24.9M 5s 2024-09-21T23:50:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30400K .......... .......... .......... .......... .......... 18% 54.5M 5s 2024-09-21T23:50:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30450K .......... .......... .......... .......... .......... 18% 22.8M 5s 2024-09-21T23:50:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30500K .......... .......... .......... .......... .......... 18% 36.2M 5s 2024-09-21T23:50:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30550K .......... .......... .......... .......... .......... 18% 47.4M 5s 2024-09-21T23:50:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30600K .......... .......... .......... .......... .......... 18% 28.4M 5s 2024-09-21T23:50:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30650K .......... .......... .......... .......... .......... 18% 20.5M 5s 2024-09-21T23:50:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30700K .......... .......... .......... .......... .......... 18% 31.7M 5s 2024-09-21T23:50:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30750K .......... .......... .......... .......... .......... 18% 23.5M 5s 2024-09-21T23:50:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30800K .......... .......... .......... .......... .......... 18% 30.6M 5s 2024-09-21T23:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30850K .......... .......... .......... .......... .......... 18% 24.8M 5s 2024-09-21T23:50:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30900K .......... .......... .......... .......... .......... 18% 55.6M 5s 2024-09-21T23:50:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 30950K .......... .......... .......... .......... .......... 18% 22.1M 5s 2024-09-21T23:50:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31000K .......... .......... .......... .......... .......... 18% 35.1M 5s 2024-09-21T23:50:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31050K .......... .......... .......... .......... .......... 18% 26.7M 5s 2024-09-21T23:50:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31100K .......... .......... .......... .......... .......... 18% 47.1M 5s 2024-09-21T23:50:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31150K .......... .......... .......... .......... .......... 18% 23.6M 5s 2024-09-21T23:50:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31200K .......... .......... .......... .......... .......... 18% 24.7M 5s 2024-09-21T23:50:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31250K .......... .......... .......... .......... .......... 18% 34.9M 5s 2024-09-21T23:50:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31300K .......... .......... .......... .......... .......... 18% 25.4M 5s 2024-09-21T23:50:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31350K .......... .......... .......... .......... .......... 18% 23.1M 5s 2024-09-21T23:50:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31400K .......... .......... .......... .......... .......... 18% 56.2M 5s 2024-09-21T23:50:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31450K .......... .......... .......... .......... .......... 18% 33.1M 5s 2024-09-21T23:50:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31500K .......... .......... .......... .......... .......... 18% 25.6M 5s 2024-09-21T23:50:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31550K .......... .......... .......... .......... .......... 18% 23.0M 5s 2024-09-21T23:50:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31600K .......... .......... .......... .......... .......... 18% 25.9M 5s 2024-09-21T23:50:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31650K .......... .......... .......... .......... .......... 18% 49.2M 5s 2024-09-21T23:50:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31700K .......... .......... .......... .......... .......... 18% 25.4M 5s 2024-09-21T23:50:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31750K .......... .......... .......... .......... .......... 18% 32.3M 5s 2024-09-21T23:50:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31800K .......... .......... .......... .......... .......... 18% 24.8M 5s 2024-09-21T23:50:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31850K .......... .......... .......... .......... .......... 18% 52.6M 5s 2024-09-21T23:50:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31900K .......... .......... .......... .......... .......... 18% 24.3M 5s 2024-09-21T23:50:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 31950K .......... .......... .......... .......... .......... 18% 24.4M 5s 2024-09-21T23:50:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32000K .......... .......... .......... .......... .......... 19% 23.5M 5s 2024-09-21T23:50:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32050K .......... .......... .......... .......... .......... 19% 36.1M 5s 2024-09-21T23:50:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32100K .......... .......... .......... .......... .......... 19% 35.8M 5s 2024-09-21T23:50:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32150K .......... .......... .......... .......... .......... 19% 31.2M 5s 2024-09-21T23:50:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32200K .......... .......... .......... .......... .......... 19% 24.1M 5s 2024-09-21T23:50:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32250K .......... .......... .......... .......... .......... 19% 23.5M 5s 2024-09-21T23:50:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32300K .......... .......... .......... .......... .......... 19% 39.6M 5s 2024-09-21T23:50:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32350K .......... .......... .......... .......... .......... 19% 23.8M 5s 2024-09-21T23:50:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32400K .......... .......... .......... .......... .......... 19% 46.6M 5s 2024-09-21T23:50:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32450K .......... .......... .......... .......... .......... 19% 25.0M 5s 2024-09-21T23:50:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32500K .......... .......... .......... .......... .......... 19% 31.2M 5s 2024-09-21T23:50:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32550K .......... .......... .......... .......... .......... 19% 25.0M 5s 2024-09-21T23:50:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32600K .......... .......... .......... .......... .......... 19% 25.4M 5s 2024-09-21T23:50:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32650K .......... .......... .......... .......... .......... 19% 56.2M 5s 2024-09-21T23:50:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32700K .......... .......... .......... .......... .......... 19% 33.6M 5s 2024-09-21T23:50:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32750K .......... .......... .......... .......... .......... 19% 20.4M 5s 2024-09-21T23:50:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32800K .......... .......... .......... .......... .......... 19% 31.7M 5s 2024-09-21T23:50:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32850K .......... .......... .......... .......... .......... 19% 24.3M 5s 2024-09-21T23:50:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32900K .......... .......... .......... .......... .......... 19% 60.6M 5s 2024-09-21T23:50:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 32950K .......... .......... .......... .......... .......... 19% 21.2M 5s 2024-09-21T23:50:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33000K .......... .......... .......... .......... .......... 19% 40.8M 5s 2024-09-21T23:50:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33050K .......... .......... .......... .......... .......... 19% 24.0M 5s 2024-09-21T23:50:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33100K .......... .......... .......... .......... .......... 19% 24.3M 5s 2024-09-21T23:50:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33150K .......... .......... .......... .......... .......... 19% 23.3M 5s 2024-09-21T23:50:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33200K .......... .......... .......... .......... .......... 19% 52.3M 5s 2024-09-21T23:50:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33250K .......... .......... .......... .......... .......... 19% 23.7M 5s 2024-09-21T23:50:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33300K .......... .......... .......... .......... .......... 19% 35.4M 5s 2024-09-21T23:50:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33350K .......... .......... .......... .......... .......... 19% 24.6M 5s 2024-09-21T23:50:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33400K .......... .......... .......... .......... .......... 19% 50.1M 5s 2024-09-21T23:50:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33450K .......... .......... .......... .......... .......... 19% 23.7M 5s 2024-09-21T23:50:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33500K .......... .......... .......... .......... .......... 19% 39.5M 5s 2024-09-21T23:50:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33550K .......... .......... .......... .......... .......... 19% 17.7M 5s 2024-09-21T23:50:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33600K .......... .......... .......... .......... .......... 19% 47.7M 5s 2024-09-21T23:50:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33650K .......... .......... .......... .......... .......... 19% 21.2M 5s 2024-09-21T23:50:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33700K .......... .......... .......... .......... .......... 20% 51.6M 5s 2024-09-21T23:50:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33750K .......... .......... .......... .......... .......... 20% 30.8M 5s 2024-09-21T23:50:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33800K .......... .......... .......... .......... .......... 20% 25.0M 5s 2024-09-21T23:50:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33850K .......... .......... .......... .......... .......... 20% 25.2M 5s 2024-09-21T23:50:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33900K .......... .......... .......... .......... .......... 20% 60.0M 5s 2024-09-21T23:50:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 33950K .......... .......... .......... .......... .......... 20% 22.4M 5s 2024-09-21T23:50:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34000K .......... .......... .......... .......... .......... 20% 30.9M 5s 2024-09-21T23:50:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34050K .......... .......... .......... .......... .......... 20% 27.8M 5s 2024-09-21T23:50:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34100K .......... .......... .......... .......... .......... 20% 24.5M 5s 2024-09-21T23:50:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34150K .......... .......... .......... .......... .......... 20% 59.4M 5s 2024-09-21T23:50:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34200K .......... .......... .......... .......... .......... 20% 22.4M 5s 2024-09-21T23:50:36.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34250K .......... .......... .......... .......... .......... 20% 23.6M 5s 2024-09-21T23:50:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34300K .......... .......... .......... .......... .......... 20% 36.2M 5s 2024-09-21T23:50:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34350K .......... .......... .......... .......... .......... 20% 22.4M 5s 2024-09-21T23:50:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34400K .......... .......... .......... .......... .......... 20% 25.0M 5s 2024-09-21T23:50:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34450K .......... .......... .......... .......... .......... 20% 48.9M 5s 2024-09-21T23:50:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34500K .......... .......... .......... .......... .......... 20% 25.8M 5s 2024-09-21T23:50:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34550K .......... .......... .......... .......... .......... 20% 32.9M 5s 2024-09-21T23:50:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34600K .......... .......... .......... .......... .......... 20% 21.0M 5s 2024-09-21T23:50:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34650K .......... .......... .......... .......... .......... 20% 128M 5s 2024-09-21T23:50:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34700K .......... .......... .......... .......... .......... 20% 23.6M 5s 2024-09-21T23:50:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34750K .......... .......... .......... .......... .......... 20% 22.1M 5s 2024-09-21T23:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34800K .......... .......... .......... .......... .......... 20% 33.7M 5s 2024-09-21T23:50:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34850K .......... .......... .......... .......... .......... 20% 24.7M 5s 2024-09-21T23:50:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34900K .......... .......... .......... .......... .......... 20% 26.6M 5s 2024-09-21T23:50:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 34950K .......... .......... .......... .......... .......... 20% 45.7M 5s 2024-09-21T23:50:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35000K .......... .......... .......... .......... .......... 20% 24.4M 5s 2024-09-21T23:50:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35050K .......... .......... .......... .......... .......... 20% 35.9M 5s 2024-09-21T23:50:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35100K .......... .......... .......... .......... .......... 20% 24.5M 5s 2024-09-21T23:50:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35150K .......... .......... .......... .......... .......... 20% 23.9M 5s 2024-09-21T23:50:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35200K .......... .......... .......... .......... .......... 20% 61.3M 5s 2024-09-21T23:50:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35250K .......... .......... .......... .......... .......... 20% 22.1M 5s 2024-09-21T23:50:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35300K .......... .......... .......... .......... .......... 20% 33.9M 5s 2024-09-21T23:50:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35350K .......... .......... .......... .......... .......... 21% 23.3M 5s 2024-09-21T23:50:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35400K .......... .......... .......... .......... .......... 21% 25.8M 5s 2024-09-21T23:50:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35450K .......... .......... .......... .......... .......... 21% 51.2M 5s 2024-09-21T23:50:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35500K .......... .......... .......... .......... .......... 21% 26.3M 5s 2024-09-21T23:50:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35550K .......... .......... .......... .......... .......... 21% 22.2M 5s 2024-09-21T23:50:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35600K .......... .......... .......... .......... .......... 21% 33.6M 5s 2024-09-21T23:50:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35650K .......... .......... .......... .......... .......... 21% 24.9M 5s 2024-09-21T23:50:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35700K .......... .......... .......... .......... .......... 21% 48.3M 5s 2024-09-21T23:50:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35750K .......... .......... .......... .......... .......... 21% 25.2M 5s 2024-09-21T23:50:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35800K .......... .......... .......... .......... .......... 21% 26.0M 5s 2024-09-21T23:50:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35850K .......... .......... .......... .......... .......... 21% 42.6M 5s 2024-09-21T23:50:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35900K .......... .......... .......... .......... .......... 21% 33.8M 5s 2024-09-21T23:50:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 35950K .......... .......... .......... .......... .......... 21% 20.6M 5s 2024-09-21T23:50:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36000K .......... .......... .......... .......... .......... 21% 33.5M 5s 2024-09-21T23:50:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36050K .......... .......... .......... .......... .......... 21% 31.6M 5s 2024-09-21T23:50:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36100K .......... .......... .......... .......... .......... 21% 24.6M 5s 2024-09-21T23:50:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36150K .......... .......... .......... .......... .......... 21% 24.6M 5s 2024-09-21T23:50:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36200K .......... .......... .......... .......... .......... 21% 49.4M 5s 2024-09-21T23:50:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36250K .......... .......... .......... .......... .......... 21% 25.3M 5s 2024-09-21T23:50:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36300K .......... .......... .......... .......... .......... 21% 35.9M 5s 2024-09-21T23:50:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36350K .......... .......... .......... .......... .......... 21% 23.4M 5s 2024-09-21T23:50:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36400K .......... .......... .......... .......... .......... 21% 24.8M 5s 2024-09-21T23:50:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36450K .......... .......... .......... .......... .......... 21% 23.6M 5s 2024-09-21T23:50:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36500K .......... .......... .......... .......... .......... 21% 59.1M 5s 2024-09-21T23:50:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36550K .......... .......... .......... .......... .......... 21% 32.4M 5s 2024-09-21T23:50:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36600K .......... .......... .......... .......... .......... 21% 23.5M 5s 2024-09-21T23:50:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36650K .......... .......... .......... .......... .......... 21% 25.6M 5s 2024-09-21T23:50:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36700K .......... .......... .......... .......... .......... 21% 46.7M 5s 2024-09-21T23:50:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36750K .......... .......... .......... .......... .......... 21% 25.8M 5s 2024-09-21T23:50:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36800K .......... .......... .......... .......... .......... 21% 24.7M 5s 2024-09-21T23:50:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36850K .......... .......... .......... .......... .......... 21% 34.5M 5s 2024-09-21T23:50:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36900K .......... .......... .......... .......... .......... 21% 24.6M 5s 2024-09-21T23:50:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 36950K .......... .......... .......... .......... .......... 21% 46.8M 5s 2024-09-21T23:50:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37000K .......... .......... .......... .......... .......... 21% 25.2M 5s 2024-09-21T23:50:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37050K .......... .......... .......... .......... .......... 22% 32.7M 5s 2024-09-21T23:50:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37100K .......... .......... .......... .......... .......... 22% 26.5M 5s 2024-09-21T23:50:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37150K .......... .......... .......... .......... .......... 22% 21.9M 5s 2024-09-21T23:50:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37200K .......... .......... .......... .......... .......... 22% 25.2M 5s 2024-09-21T23:50:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37250K .......... .......... .......... .......... .......... 22% 50.7M 5s 2024-09-21T23:50:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37300K .......... .......... .......... .......... .......... 22% 24.0M 5s 2024-09-21T23:50:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37350K .......... .......... .......... .......... .......... 22% 37.8M 5s 2024-09-21T23:50:36.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37400K .......... .......... .......... .......... .......... 22% 38.0M 5s 2024-09-21T23:50:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37450K .......... .......... .......... .......... .......... 22% 30.6M 5s 2024-09-21T23:50:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37500K .......... .......... .......... .......... .......... 22% 23.7M 5s 2024-09-21T23:50:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37550K .......... .......... .......... .......... .......... 22% 23.9M 5s 2024-09-21T23:50:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37600K .......... .......... .......... .......... .......... 22% 32.7M 5s 2024-09-21T23:50:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37650K .......... .......... .......... .......... .......... 22% 24.6M 5s 2024-09-21T23:50:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37700K .......... .......... .......... .......... .......... 22% 25.4M 5s 2024-09-21T23:50:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37750K .......... .......... .......... .......... .......... 22% 51.4M 5s 2024-09-21T23:50:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37800K .......... .......... .......... .......... .......... 22% 23.3M 5s 2024-09-21T23:50:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37850K .......... .......... .......... .......... .......... 22% 36.4M 5s 2024-09-21T23:50:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37900K .......... .......... .......... .......... .......... 22% 25.1M 5s 2024-09-21T23:50:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 37950K .......... .......... .......... .......... .......... 22% 23.6M 5s 2024-09-21T23:50:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38000K .......... .......... .......... .......... .......... 22% 50.6M 5s 2024-09-21T23:50:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38050K .......... .......... .......... .......... .......... 22% 24.0M 5s 2024-09-21T23:50:36.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38100K .......... .......... .......... .......... .......... 22% 34.8M 5s 2024-09-21T23:50:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38150K .......... .......... .......... .......... .......... 22% 22.7M 5s 2024-09-21T23:50:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38200K .......... .......... .......... .......... .......... 22% 23.9M 5s 2024-09-21T23:50:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38250K .......... .......... .......... .......... .......... 22% 56.7M 5s 2024-09-21T23:50:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38300K .......... .......... .......... .......... .......... 22% 33.0M 5s 2024-09-21T23:50:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38350K .......... .......... .......... .......... .......... 22% 18.7M 5s 2024-09-21T23:50:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38400K .......... .......... .......... .......... .......... 22% 31.4M 5s 2024-09-21T23:50:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38450K .......... .......... .......... .......... .......... 22% 26.7M 5s 2024-09-21T23:50:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38500K .......... .......... .......... .......... .......... 22% 49.6M 5s 2024-09-21T23:50:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38550K .......... .......... .......... .......... .......... 22% 25.0M 5s 2024-09-21T23:50:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38600K .......... .......... .......... .......... .......... 22% 35.4M 5s 2024-09-21T23:50:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38650K .......... .......... .......... .......... .......... 22% 24.9M 5s 2024-09-21T23:50:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38700K .......... .......... .......... .......... .......... 22% 53.0M 5s 2024-09-21T23:50:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38750K .......... .......... .......... .......... .......... 23% 15.5M 5s 2024-09-21T23:50:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38800K .......... .......... .......... .......... .......... 23% 52.3M 5s 2024-09-21T23:50:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38850K .......... .......... .......... .......... .......... 23% 34.0M 5s 2024-09-21T23:50:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38900K .......... .......... .......... .......... .......... 23% 28.5M 5s 2024-09-21T23:50:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 38950K .......... .......... .......... .......... .......... 23% 24.2M 5s 2024-09-21T23:50:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39000K .......... .......... .......... .......... .......... 23% 36.6M 5s 2024-09-21T23:50:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39050K .......... .......... .......... .......... .......... 23% 24.4M 5s 2024-09-21T23:50:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39100K .......... .......... .......... .......... .......... 23% 36.2M 5s 2024-09-21T23:50:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39150K .......... .......... .......... .......... .......... 23% 15.3M 5s 2024-09-21T23:50:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39200K .......... .......... .......... .......... .......... 23% 63.8M 5s 2024-09-21T23:50:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39250K .......... .......... .......... .......... .......... 23% 46.3M 5s 2024-09-21T23:50:36.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39300K .......... .......... .......... .......... .......... 23% 25.7M 5s 2024-09-21T23:50:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39350K .......... .......... .......... .......... .......... 23% 30.1M 5s 2024-09-21T23:50:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39400K .......... .......... .......... .......... .......... 23% 25.0M 5s 2024-09-21T23:50:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39450K .......... .......... .......... .......... .......... 23% 23.8M 5s 2024-09-21T23:50:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39500K .......... .......... .......... .......... .......... 23% 51.3M 5s 2024-09-21T23:50:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39550K .......... .......... .......... .......... .......... 23% 24.8M 5s 2024-09-21T23:50:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39600K .......... .......... .......... .......... .......... 23% 24.4M 5s 2024-09-21T23:50:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39650K .......... .......... .......... .......... .......... 23% 35.0M 5s 2024-09-21T23:50:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39700K .......... .......... .......... .......... .......... 23% 28.9M 5s 2024-09-21T23:50:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39750K .......... .......... .......... .......... .......... 23% 40.5M 5s 2024-09-21T23:50:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39800K .......... .......... .......... .......... .......... 23% 23.9M 5s 2024-09-21T23:50:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39850K .......... .......... .......... .......... .......... 23% 37.9M 5s 2024-09-21T23:50:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39900K .......... .......... .......... .......... .......... 23% 22.6M 5s 2024-09-21T23:50:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 39950K .......... .......... .......... .......... .......... 23% 24.1M 5s 2024-09-21T23:50:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40000K .......... .......... .......... .......... .......... 23% 25.3M 5s 2024-09-21T23:50:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40050K .......... .......... .......... .......... .......... 23% 49.0M 5s 2024-09-21T23:50:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40100K .......... .......... .......... .......... .......... 23% 25.5M 5s 2024-09-21T23:50:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40150K .......... .......... .......... .......... .......... 23% 33.3M 5s 2024-09-21T23:50:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40200K .......... .......... .......... .......... .......... 23% 24.2M 5s 2024-09-21T23:50:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40250K .......... .......... .......... .......... .......... 23% 51.3M 5s 2024-09-21T23:50:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40300K .......... .......... .......... .......... .......... 23% 25.2M 5s 2024-09-21T23:50:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40350K .......... .......... .......... .......... .......... 23% 22.9M 5s 2024-09-21T23:50:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40400K .......... .......... .......... .......... .......... 23% 38.5M 5s 2024-09-21T23:50:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40450K .......... .......... .......... .......... .......... 24% 22.9M 5s 2024-09-21T23:50:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40500K .......... .......... .......... .......... .......... 24% 25.2M 5s 2024-09-21T23:50:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40550K .......... .......... .......... .......... .......... 24% 52.7M 5s 2024-09-21T23:50:36.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40600K .......... .......... .......... .......... .......... 24% 23.6M 5s 2024-09-21T23:50:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40650K .......... .......... .......... .......... .......... 24% 36.5M 5s 2024-09-21T23:50:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40700K .......... .......... .......... .......... .......... 24% 24.4M 5s 2024-09-21T23:50:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40750K .......... .......... .......... .......... .......... 24% 23.5M 5s 2024-09-21T23:50:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40800K .......... .......... .......... .......... .......... 24% 48.4M 5s 2024-09-21T23:50:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40850K .......... .......... .......... .......... .......... 24% 25.7M 5s 2024-09-21T23:50:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40900K .......... .......... .......... .......... .......... 24% 36.0M 5s 2024-09-21T23:50:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 40950K .......... .......... .......... .......... .......... 24% 24.2M 5s 2024-09-21T23:50:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41000K .......... .......... .......... .......... .......... 24% 24.1M 5s 2024-09-21T23:50:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41050K .......... .......... .......... .......... .......... 24% 54.5M 5s 2024-09-21T23:50:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41100K .......... .......... .......... .......... .......... 24% 31.7M 5s 2024-09-21T23:50:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41150K .......... .......... .......... .......... .......... 24% 18.9M 5s 2024-09-21T23:50:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41200K .......... .......... .......... .......... .......... 24% 33.0M 5s 2024-09-21T23:50:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41250K .......... .......... .......... .......... .......... 24% 25.5M 5s 2024-09-21T23:50:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41300K .......... .......... .......... .......... .......... 24% 52.2M 5s 2024-09-21T23:50:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41350K .......... .......... .......... .......... .......... 24% 24.3M 5s 2024-09-21T23:50:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41400K .......... .......... .......... .......... .......... 24% 33.8M 5s 2024-09-21T23:50:36.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41450K .......... .......... .......... .......... .......... 24% 24.9M 5s 2024-09-21T23:50:36.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41500K .......... .......... .......... .......... .......... 24% 51.1M 5s 2024-09-21T23:50:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41550K .......... .......... .......... .......... .......... 24% 15.3M 5s 2024-09-21T23:50:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41600K .......... .......... .......... .......... .......... 24% 63.2M 5s 2024-09-21T23:50:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41650K .......... .......... .......... .......... .......... 24% 30.2M 5s 2024-09-21T23:50:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41700K .......... .......... .......... .......... .......... 24% 25.3M 5s 2024-09-21T23:50:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41750K .......... .......... .......... .......... .......... 24% 24.1M 5s 2024-09-21T23:50:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41800K .......... .......... .......... .......... .......... 24% 55.2M 5s 2024-09-21T23:50:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41850K .......... .......... .......... .......... .......... 24% 24.8M 5s 2024-09-21T23:50:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41900K .......... .......... .......... .......... .......... 24% 34.0M 5s 2024-09-21T23:50:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 41950K .......... .......... .......... .......... .......... 24% 24.3M 5s 2024-09-21T23:50:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42000K .......... .......... .......... .......... .......... 24% 25.0M 5s 2024-09-21T23:50:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42050K .......... .......... .......... .......... .......... 24% 47.9M 5s 2024-09-21T23:50:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42100K .......... .......... .......... .......... .......... 25% 24.0M 5s 2024-09-21T23:50:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42150K .......... .......... .......... .......... .......... 25% 35.5M 5s 2024-09-21T23:50:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42200K .......... .......... .......... .......... .......... 25% 23.6M 5s 2024-09-21T23:50:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42250K .......... .......... .......... .......... .......... 25% 25.0M 5s 2024-09-21T23:50:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42300K .......... .......... .......... .......... .......... 25% 58.9M 5s 2024-09-21T23:50:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42350K .......... .......... .......... .......... .......... 25% 23.3M 5s 2024-09-21T23:50:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42400K .......... .......... .......... .......... .......... 25% 24.3M 5s 2024-09-21T23:50:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42450K .......... .......... .......... .......... .......... 25% 34.7M 5s 2024-09-21T23:50:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42500K .......... .......... .......... .......... .......... 25% 30.3M 5s 2024-09-21T23:50:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42550K .......... .......... .......... .......... .......... 25% 31.5M 5s 2024-09-21T23:50:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42600K .......... .......... .......... .......... .......... 25% 27.9M 5s 2024-09-21T23:50:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42650K .......... .......... .......... .......... .......... 25% 34.2M 5s 2024-09-21T23:50:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42700K .......... .......... .......... .......... .......... 25% 23.0M 5s 2024-09-21T23:50:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42750K .......... .......... .......... .......... .......... 25% 24.9M 5s 2024-09-21T23:50:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42800K .......... .......... .......... .......... .......... 25% 589K 5s 2024-09-21T23:50:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42850K .......... .......... .......... .......... .......... 25% 123M 5s 2024-09-21T23:50:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42900K .......... .......... .......... .......... .......... 25% 129M 5s 2024-09-21T23:50:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 42950K .......... .......... .......... .......... .......... 25% 108M 5s 2024-09-21T23:50:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43000K .......... .......... .......... .......... .......... 25% 122M 5s 2024-09-21T23:50:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43050K .......... .......... .......... .......... .......... 25% 126M 5s 2024-09-21T23:50:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43100K .......... .......... .......... .......... .......... 25% 125M 5s 2024-09-21T23:50:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43150K .......... .......... .......... .......... .......... 25% 96.1M 5s 2024-09-21T23:50:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43200K .......... .......... .......... .......... .......... 25% 123M 5s 2024-09-21T23:50:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43250K .......... .......... .......... .......... .......... 25% 128M 5s 2024-09-21T23:50:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43300K .......... .......... .......... .......... .......... 25% 128M 5s 2024-09-21T23:50:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43350K .......... .......... .......... .......... .......... 25% 112M 5s 2024-09-21T23:50:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43400K .......... .......... .......... .......... .......... 25% 130M 5s 2024-09-21T23:50:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43450K .......... .......... .......... .......... .......... 25% 120M 5s 2024-09-21T23:50:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43500K .......... .......... .......... .......... .......... 25% 129M 5s 2024-09-21T23:50:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43550K .......... .......... .......... .......... .......... 25% 103M 5s 2024-09-21T23:50:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43600K .......... .......... .......... .......... .......... 25% 5.70M 5s 2024-09-21T23:50:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43650K .......... .......... .......... .......... .......... 25% 97.9M 5s 2024-09-21T23:50:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43700K .......... .......... .......... .......... .......... 25% 76.4M 5s 2024-09-21T23:50:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43750K .......... .......... .......... .......... .......... 25% 116M 5s 2024-09-21T23:50:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43800K .......... .......... .......... .......... .......... 26% 106M 5s 2024-09-21T23:50:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43850K .......... .......... .......... .......... .......... 26% 123M 5s 2024-09-21T23:50:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43900K .......... .......... .......... .......... .......... 26% 131M 5s 2024-09-21T23:50:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 43950K .......... .......... .......... .......... .......... 26% 84.2M 5s 2024-09-21T23:50:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44000K .......... .......... .......... .......... .......... 26% 126M 5s 2024-09-21T23:50:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44050K .......... .......... .......... .......... .......... 26% 121M 5s 2024-09-21T23:50:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44100K .......... .......... .......... .......... .......... 26% 123M 5s 2024-09-21T23:50:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44150K .......... .......... .......... .......... .......... 26% 118M 5s 2024-09-21T23:50:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44200K .......... .......... .......... .......... .......... 26% 129M 5s 2024-09-21T23:50:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44250K .......... .......... .......... .......... .......... 26% 127M 5s 2024-09-21T23:50:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44300K .......... .......... .......... .......... .......... 26% 131M 5s 2024-09-21T23:50:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44350K .......... .......... .......... .......... .......... 26% 104M 5s 2024-09-21T23:50:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44400K .......... .......... .......... .......... .......... 26% 131M 5s 2024-09-21T23:50:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44450K .......... .......... .......... .......... .......... 26% 124M 5s 2024-09-21T23:50:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44500K .......... .......... .......... .......... .......... 26% 132M 5s 2024-09-21T23:50:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44550K .......... .......... .......... .......... .......... 26% 109M 5s 2024-09-21T23:50:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44600K .......... .......... .......... .......... .......... 26% 123M 5s 2024-09-21T23:50:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44650K .......... .......... .......... .......... .......... 26% 126M 5s 2024-09-21T23:50:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44700K .......... .......... .......... .......... .......... 26% 124M 5s 2024-09-21T23:50:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44750K .......... .......... .......... .......... .......... 26% 110M 5s 2024-09-21T23:50:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44800K .......... .......... .......... .......... .......... 26% 125M 5s 2024-09-21T23:50:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44850K .......... .......... .......... .......... .......... 26% 128M 5s 2024-09-21T23:50:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44900K .......... .......... .......... .......... .......... 26% 127M 5s 2024-09-21T23:50:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 44950K .......... .......... .......... .......... .......... 26% 117M 5s 2024-09-21T23:50:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45000K .......... .......... .......... .......... .......... 26% 128M 5s 2024-09-21T23:50:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45050K .......... .......... .......... .......... .......... 26% 121M 5s 2024-09-21T23:50:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45100K .......... .......... .......... .......... .......... 26% 131M 5s 2024-09-21T23:50:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45150K .......... .......... .......... .......... .......... 26% 89.3M 5s 2024-09-21T23:50:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45200K .......... .......... .......... .......... .......... 26% 128M 5s 2024-09-21T23:50:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45250K .......... .......... .......... .......... .......... 26% 129M 5s 2024-09-21T23:50:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45300K .......... .......... .......... .......... .......... 26% 120M 5s 2024-09-21T23:50:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45350K .......... .......... .......... .......... .......... 26% 116M 5s 2024-09-21T23:50:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45400K .......... .......... .......... .......... .......... 26% 123M 4s 2024-09-21T23:50:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45450K .......... .......... .......... .......... .......... 26% 129M 4s 2024-09-21T23:50:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45500K .......... .......... .......... .......... .......... 27% 122M 4s 2024-09-21T23:50:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45550K .......... .......... .......... .......... .......... 27% 103M 4s 2024-09-21T23:50:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45600K .......... .......... .......... .......... .......... 27% 130M 4s 2024-09-21T23:50:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45650K .......... .......... .......... .......... .......... 27% 121M 4s 2024-09-21T23:50:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45700K .......... .......... .......... .......... .......... 27% 127M 4s 2024-09-21T23:50:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45750K .......... .......... .......... .......... .......... 27% 108M 4s 2024-09-21T23:50:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45800K .......... .......... .......... .......... .......... 27% 132M 4s 2024-09-21T23:50:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45850K .......... .......... .......... .......... .......... 27% 10.7M 4s 2024-09-21T23:50:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45900K .......... .......... .......... .......... .......... 27% 94.2M 4s 2024-09-21T23:50:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 45950K .......... .......... .......... .......... .......... 27% 89.4M 4s 2024-09-21T23:50:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46000K .......... .......... .......... .......... .......... 27% 118M 4s 2024-09-21T23:50:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46050K .......... .......... .......... .......... .......... 27% 110M 4s 2024-09-21T23:50:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46100K .......... .......... .......... .......... .......... 27% 109M 4s 2024-09-21T23:50:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46150K .......... .......... .......... .......... .......... 27% 111M 4s 2024-09-21T23:50:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46200K .......... .......... .......... .......... .......... 27% 117M 4s 2024-09-21T23:50:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46250K .......... .......... .......... .......... .......... 27% 76.7M 4s 2024-09-21T23:50:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46300K .......... .......... .......... .......... .......... 27% 24.0M 4s 2024-09-21T23:50:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46350K .......... .......... .......... .......... .......... 27% 23.8M 4s 2024-09-21T23:50:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46400K .......... .......... .......... .......... .......... 27% 43.8M 4s 2024-09-21T23:50:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46450K .......... .......... .......... .......... .......... 27% 24.6M 4s 2024-09-21T23:50:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46500K .......... .......... .......... .......... .......... 27% 23.6M 4s 2024-09-21T23:50:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46550K .......... .......... .......... .......... .......... 27% 38.6M 4s 2024-09-21T23:50:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46600K .......... .......... .......... .......... .......... 27% 38.0M 4s 2024-09-21T23:50:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46650K .......... .......... .......... .......... .......... 27% 26.4M 4s 2024-09-21T23:50:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46700K .......... .......... .......... .......... .......... 27% 24.5M 4s 2024-09-21T23:50:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46750K .......... .......... .......... .......... .......... 27% 22.0M 4s 2024-09-21T23:50:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46800K .......... .......... .......... .......... .......... 27% 45.5M 4s 2024-09-21T23:50:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46850K .......... .......... .......... .......... .......... 27% 24.4M 4s 2024-09-21T23:50:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46900K .......... .......... .......... .......... .......... 27% 44.9M 4s 2024-09-21T23:50:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 46950K .......... .......... .......... .......... .......... 27% 24.1M 4s 2024-09-21T23:50:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47000K .......... .......... .......... .......... .......... 27% 39.1M 4s 2024-09-21T23:50:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47050K .......... .......... .......... .......... .......... 27% 24.1M 4s 2024-09-21T23:50:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47100K .......... .......... .......... .......... .......... 27% 40.6M 4s 2024-09-21T23:50:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47150K .......... .......... .......... .......... .......... 28% 23.7M 4s 2024-09-21T23:50:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47200K .......... .......... .......... .......... .......... 28% 27.1M 4s 2024-09-21T23:50:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47250K .......... .......... .......... .......... .......... 28% 27.1M 4s 2024-09-21T23:50:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47300K .......... .......... .......... .......... .......... 28% 25.6M 4s 2024-09-21T23:50:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47350K .......... .......... .......... .......... .......... 28% 23.1M 4s 2024-09-21T23:50:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47400K .......... .......... .......... .......... .......... 28% 59.6M 4s 2024-09-21T23:50:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47450K .......... .......... .......... .......... .......... 28% 31.7M 4s 2024-09-21T23:50:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47500K .......... .......... .......... .......... .......... 28% 22.3M 4s 2024-09-21T23:50:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47550K .......... .......... .......... .......... .......... 28% 23.3M 4s 2024-09-21T23:50:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47600K .......... .......... .......... .......... .......... 28% 24.6M 4s 2024-09-21T23:50:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47650K .......... .......... .......... .......... .......... 28% 108M 4s 2024-09-21T23:50:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47700K .......... .......... .......... .......... .......... 28% 23.8M 4s 2024-09-21T23:50:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47750K .......... .......... .......... .......... .......... 28% 22.2M 4s 2024-09-21T23:50:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47800K .......... .......... .......... .......... .......... 28% 25.4M 4s 2024-09-21T23:50:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47850K .......... .......... .......... .......... .......... 28% 38.8M 4s 2024-09-21T23:50:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47900K .......... .......... .......... .......... .......... 28% 44.0M 4s 2024-09-21T23:50:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 47950K .......... .......... .......... .......... .......... 28% 25.2M 4s 2024-09-21T23:50:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48000K .......... .......... .......... .......... .......... 28% 28.6M 4s 2024-09-21T23:50:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48050K .......... .......... .......... .......... .......... 28% 24.0M 4s 2024-09-21T23:50:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48100K .......... .......... .......... .......... .......... 28% 26.4M 4s 2024-09-21T23:50:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48150K .......... .......... .......... .......... .......... 28% 55.2M 4s 2024-09-21T23:50:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48200K .......... .......... .......... .......... .......... 28% 24.5M 4s 2024-09-21T23:50:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48250K .......... .......... .......... .......... .......... 28% 38.3M 4s 2024-09-21T23:50:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48300K .......... .......... .......... .......... .......... 28% 25.4M 4s 2024-09-21T23:50:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48350K .......... .......... .......... .......... .......... 28% 23.7M 4s 2024-09-21T23:50:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48400K .......... .......... .......... .......... .......... 28% 25.5M 4s 2024-09-21T23:50:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48450K .......... .......... .......... .......... .......... 28% 43.0M 4s 2024-09-21T23:50:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48500K .......... .......... .......... .......... .......... 28% 24.8M 4s 2024-09-21T23:50:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48550K .......... .......... .......... .......... .......... 28% 34.1M 4s 2024-09-21T23:50:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48600K .......... .......... .......... .......... .......... 28% 22.7M 4s 2024-09-21T23:50:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48650K .......... .......... .......... .......... .......... 28% 52.6M 4s 2024-09-21T23:50:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48700K .......... .......... .......... .......... .......... 28% 23.1M 4s 2024-09-21T23:50:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48750K .......... .......... .......... .......... .......... 28% 25.0M 4s 2024-09-21T23:50:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48800K .......... .......... .......... .......... .......... 28% 34.9M 4s 2024-09-21T23:50:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48850K .......... .......... .......... .......... .......... 29% 25.2M 4s 2024-09-21T23:50:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48900K .......... .......... .......... .......... .......... 29% 23.2M 4s 2024-09-21T23:50:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 48950K .......... .......... .......... .......... .......... 29% 52.7M 4s 2024-09-21T23:50:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49000K .......... .......... .......... .......... .......... 29% 34.7M 4s 2024-09-21T23:50:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49050K .......... .......... .......... .......... .......... 29% 24.1M 4s 2024-09-21T23:50:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49100K .......... .......... .......... .......... .......... 29% 25.5M 4s 2024-09-21T23:50:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49150K .......... .......... .......... .......... .......... 29% 23.6M 4s 2024-09-21T23:50:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49200K .......... .......... .......... .......... .......... 29% 51.9M 4s 2024-09-21T23:50:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49250K .......... .......... .......... .......... .......... 29% 23.6M 4s 2024-09-21T23:50:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49300K .......... .......... .......... .......... .......... 29% 38.8M 4s 2024-09-21T23:50:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49350K .......... .......... .......... .......... .......... 29% 24.6M 4s 2024-09-21T23:50:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49400K .......... .......... .......... .......... .......... 29% 47.1M 4s 2024-09-21T23:50:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49450K .......... .......... .......... .......... .......... 29% 25.1M 4s 2024-09-21T23:50:37.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49500K .......... .......... .......... .......... .......... 29% 32.9M 4s 2024-09-21T23:50:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49550K .......... .......... .......... .......... .......... 29% 24.4M 4s 2024-09-21T23:50:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49600K .......... .......... .......... .......... .......... 29% 24.6M 4s 2024-09-21T23:50:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49650K .......... .......... .......... .......... .......... 29% 24.0M 4s 2024-09-21T23:50:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49700K .......... .......... .......... .......... .......... 29% 45.9M 4s 2024-09-21T23:50:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49750K .......... .......... .......... .......... .......... 29% 31.6M 4s 2024-09-21T23:50:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49800K .......... .......... .......... .......... .......... 29% 27.2M 4s 2024-09-21T23:50:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49850K .......... .......... .......... .......... .......... 29% 24.5M 4s 2024-09-21T23:50:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49900K .......... .......... .......... .......... .......... 29% 49.9M 4s 2024-09-21T23:50:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 49950K .......... .......... .......... .......... .......... 29% 24.1M 4s 2024-09-21T23:50:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50000K .......... .......... .......... .......... .......... 29% 35.3M 4s 2024-09-21T23:50:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50050K .......... .......... .......... .......... .......... 29% 23.5M 4s 2024-09-21T23:50:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50100K .......... .......... .......... .......... .......... 29% 25.5M 4s 2024-09-21T23:50:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50150K .......... .......... .......... .......... .......... 29% 24.5M 4s 2024-09-21T23:50:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50200K .......... .......... .......... .......... .......... 29% 48.8M 4s 2024-09-21T23:50:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50250K .......... .......... .......... .......... .......... 29% 24.5M 4s 2024-09-21T23:50:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50300K .......... .......... .......... .......... .......... 29% 36.1M 4s 2024-09-21T23:50:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50350K .......... .......... .......... .......... .......... 29% 24.1M 4s 2024-09-21T23:50:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50400K .......... .......... .......... .......... .......... 29% 24.8M 4s 2024-09-21T23:50:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50450K .......... .......... .......... .......... .......... 29% 49.6M 4s 2024-09-21T23:50:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50500K .......... .......... .......... .......... .......... 29% 24.9M 4s 2024-09-21T23:50:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50550K .......... .......... .......... .......... .......... 30% 35.7M 4s 2024-09-21T23:50:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50600K .......... .......... .......... .......... .......... 30% 24.8M 4s 2024-09-21T23:50:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50650K .......... .......... .......... .......... .......... 30% 22.6M 4s 2024-09-21T23:50:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50700K .......... .......... .......... .......... .......... 30% 56.8M 4s 2024-09-21T23:50:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50750K .......... .......... .......... .......... .......... 30% 21.9M 4s 2024-09-21T23:50:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50800K .......... .......... .......... .......... .......... 30% 24.6M 4s 2024-09-21T23:50:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50850K .......... .......... .......... .......... .......... 30% 38.6M 4s 2024-09-21T23:50:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50900K .......... .......... .......... .......... .......... 30% 25.1M 4s 2024-09-21T23:50:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 50950K .......... .......... .......... .......... .......... 30% 44.2M 4s 2024-09-21T23:50:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51000K .......... .......... .......... .......... .......... 30% 25.1M 4s 2024-09-21T23:50:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51050K .......... .......... .......... .......... .......... 30% 37.5M 4s 2024-09-21T23:50:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51100K .......... .......... .......... .......... .......... 30% 23.8M 4s 2024-09-21T23:50:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51150K .......... .......... .......... .......... .......... 30% 22.4M 4s 2024-09-21T23:50:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51200K .......... .......... .......... .......... .......... 30% 23.9M 4s 2024-09-21T23:50:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51250K .......... .......... .......... .......... .......... 30% 56.2M 4s 2024-09-21T23:50:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51300K .......... .......... .......... .......... .......... 30% 23.2M 4s 2024-09-21T23:50:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51350K .......... .......... .......... .......... .......... 30% 34.1M 4s 2024-09-21T23:50:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51400K .......... .......... .......... .......... .......... 30% 24.1M 4s 2024-09-21T23:50:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51450K .......... .......... .......... .......... .......... 30% 53.0M 4s 2024-09-21T23:50:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51500K .......... .......... .......... .......... .......... 30% 24.5M 4s 2024-09-21T23:50:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51550K .......... .......... .......... .......... .......... 30% 22.9M 4s 2024-09-21T23:50:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51600K .......... .......... .......... .......... .......... 30% 37.7M 4s 2024-09-21T23:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51650K .......... .......... .......... .......... .......... 30% 24.7M 4s 2024-09-21T23:50:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51700K .......... .......... .......... .......... .......... 30% 48.0M 4s 2024-09-21T23:50:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51750K .......... .......... .......... .......... .......... 30% 23.8M 4s 2024-09-21T23:50:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51800K .......... .......... .......... .......... .......... 30% 34.1M 4s 2024-09-21T23:50:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51850K .......... .......... .......... .......... .......... 30% 25.1M 4s 2024-09-21T23:50:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51900K .......... .......... .......... .......... .......... 30% 37.2M 4s 2024-09-21T23:50:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 51950K .......... .......... .......... .......... .......... 30% 18.2M 4s 2024-09-21T23:50:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52000K .......... .......... .......... .......... .......... 30% 46.8M 4s 2024-09-21T23:50:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52050K .......... .......... .......... .......... .......... 30% 25.4M 4s 2024-09-21T23:50:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52100K .......... .......... .......... .......... .......... 30% 36.3M 4s 2024-09-21T23:50:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52150K .......... .......... .......... .......... .......... 30% 24.2M 4s 2024-09-21T23:50:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52200K .......... .......... .......... .......... .......... 30% 47.3M 4s 2024-09-21T23:50:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52250K .......... .......... .......... .......... .......... 31% 24.7M 4s 2024-09-21T23:50:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52300K .......... .......... .......... .......... .......... 31% 37.9M 4s 2024-09-21T23:50:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52350K .......... .......... .......... .......... .......... 31% 17.7M 4s 2024-09-21T23:50:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52400K .......... .......... .......... .......... .......... 31% 26.8M 4s 2024-09-21T23:50:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52450K .......... .......... .......... .......... .......... 31% 31.9M 4s 2024-09-21T23:50:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52500K .......... .......... .......... .......... .......... 31% 48.2M 4s 2024-09-21T23:50:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52550K .......... .......... .......... .......... .......... 31% 24.3M 4s 2024-09-21T23:50:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52600K .......... .......... .......... .......... .......... 31% 38.2M 4s 2024-09-21T23:50:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52650K .......... .......... .......... .......... .......... 31% 23.7M 4s 2024-09-21T23:50:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52700K .......... .......... .......... .......... .......... 31% 47.8M 4s 2024-09-21T23:50:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52750K .......... .......... .......... .......... .......... 31% 23.4M 4s 2024-09-21T23:50:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52800K .......... .......... .......... .......... .......... 31% 25.9M 4s 2024-09-21T23:50:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52850K .......... .......... .......... .......... .......... 31% 36.0M 4s 2024-09-21T23:50:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52900K .......... .......... .......... .......... .......... 31% 21.3M 4s 2024-09-21T23:50:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 52950K .......... .......... .......... .......... .......... 31% 28.0M 4s 2024-09-21T23:50:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53000K .......... .......... .......... .......... .......... 31% 55.1M 4s 2024-09-21T23:50:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53050K .......... .......... .......... .......... .......... 31% 23.4M 4s 2024-09-21T23:50:37.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53100K .......... .......... .......... .......... .......... 31% 34.0M 4s 2024-09-21T23:50:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53150K .......... .......... .......... .......... .......... 31% 24.7M 4s 2024-09-21T23:50:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53200K .......... .......... .......... .......... .......... 31% 24.7M 4s 2024-09-21T23:50:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53250K .......... .......... .......... .......... .......... 31% 47.9M 4s 2024-09-21T23:50:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53300K .......... .......... .......... .......... .......... 31% 25.2M 4s 2024-09-21T23:50:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53350K .......... .......... .......... .......... .......... 31% 34.9M 4s 2024-09-21T23:50:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53400K .......... .......... .......... .......... .......... 31% 24.9M 4s 2024-09-21T23:50:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53450K .......... .......... .......... .......... .......... 31% 23.5M 4s 2024-09-21T23:50:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53500K .......... .......... .......... .......... .......... 31% 54.2M 4s 2024-09-21T23:50:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53550K .......... .......... .......... .......... .......... 31% 22.1M 4s 2024-09-21T23:50:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53600K .......... .......... .......... .......... .......... 31% 25.4M 4s 2024-09-21T23:50:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53650K .......... .......... .......... .......... .......... 31% 37.0M 4s 2024-09-21T23:50:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53700K .......... .......... .......... .......... .......... 31% 24.3M 4s 2024-09-21T23:50:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53750K .......... .......... .......... .......... .......... 31% 45.9M 4s 2024-09-21T23:50:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53800K .......... .......... .......... .......... .......... 31% 24.3M 4s 2024-09-21T23:50:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53850K .......... .......... .......... .......... .......... 31% 36.2M 4s 2024-09-21T23:50:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53900K .......... .......... .......... .......... .......... 32% 24.6M 4s 2024-09-21T23:50:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 53950K .......... .......... .......... .......... .......... 32% 32.5M 4s 2024-09-21T23:50:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54000K .......... .......... .......... .......... .......... 32% 19.2M 4s 2024-09-21T23:50:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54050K .......... .......... .......... .......... .......... 32% 54.1M 4s 2024-09-21T23:50:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54100K .......... .......... .......... .......... .......... 32% 33.8M 4s 2024-09-21T23:50:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54150K .......... .......... .......... .......... .......... 32% 24.5M 4s 2024-09-21T23:50:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54200K .......... .......... .......... .......... .......... 32% 24.4M 4s 2024-09-21T23:50:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54250K .......... .......... .......... .......... .......... 32% 45.9M 4s 2024-09-21T23:50:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54300K .......... .......... .......... .......... .......... 32% 25.7M 4s 2024-09-21T23:50:37.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54350K .......... .......... .......... .......... .......... 32% 24.3M 4s 2024-09-21T23:50:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54400K .......... .......... .......... .......... .......... 32% 37.5M 4s 2024-09-21T23:50:37.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54450K .......... .......... .......... .......... .......... 32% 24.2M 4s 2024-09-21T23:50:37.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54500K .......... .......... .......... .......... .......... 32% 45.7M 4s 2024-09-21T23:50:37.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54550K .......... .......... .......... .......... .......... 32% 24.5M 4s 2024-09-21T23:50:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54600K .......... .......... .......... .......... .......... 32% 32.3M 4s 2024-09-21T23:50:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54650K .......... .......... .......... .......... .......... 32% 26.1M 4s 2024-09-21T23:50:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54700K .......... .......... .......... .......... .......... 32% 24.7M 4s 2024-09-21T23:50:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54750K .......... .......... .......... .......... .......... 32% 25.1M 4s 2024-09-21T23:50:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54800K .......... .......... .......... .......... .......... 32% 43.4M 4s 2024-09-21T23:50:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54850K .......... .......... .......... .......... .......... 32% 25.3M 4s 2024-09-21T23:50:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54900K .......... .......... .......... .......... .......... 32% 37.3M 4s 2024-09-21T23:50:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 54950K .......... .......... .......... .......... .......... 32% 24.1M 4s 2024-09-21T23:50:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55000K .......... .......... .......... .......... .......... 32% 45.4M 4s 2024-09-21T23:50:37.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55050K .......... .......... .......... .......... .......... 32% 24.8M 4s 2024-09-21T23:50:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55100K .......... .......... .......... .......... .......... 32% 33.4M 4s 2024-09-21T23:50:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55150K .......... .......... .......... .......... .......... 32% 23.8M 4s 2024-09-21T23:50:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55200K .......... .......... .......... .......... .......... 32% 25.1M 4s 2024-09-21T23:50:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55250K .......... .......... .......... .......... .......... 32% 25.0M 4s 2024-09-21T23:50:37.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55300K .......... .......... .......... .......... .......... 32% 46.7M 4s 2024-09-21T23:50:37.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55350K .......... .......... .......... .......... .......... 32% 24.7M 4s 2024-09-21T23:50:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55400K .......... .......... .......... .......... .......... 32% 37.1M 4s 2024-09-21T23:50:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55450K .......... .......... .......... .......... .......... 32% 25.0M 4s 2024-09-21T23:50:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55500K .......... .......... .......... .......... .......... 32% 49.5M 4s 2024-09-21T23:50:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55550K .......... .......... .......... .......... .......... 32% 23.4M 4s 2024-09-21T23:50:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55600K .......... .......... .......... .......... .......... 33% 25.3M 4s 2024-09-21T23:50:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55650K .......... .......... .......... .......... .......... 33% 33.7M 4s 2024-09-21T23:50:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55700K .......... .......... .......... .......... .......... 33% 23.9M 4s 2024-09-21T23:50:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55750K .......... .......... .......... .......... .......... 33% 24.8M 4s 2024-09-21T23:50:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55800K .......... .......... .......... .......... .......... 33% 53.2M 4s 2024-09-21T23:50:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55850K .......... .......... .......... .......... .......... 33% 32.7M 4s 2024-09-21T23:50:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55900K .......... .......... .......... .......... .......... 33% 25.7M 4s 2024-09-21T23:50:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 55950K .......... .......... .......... .......... .......... 33% 24.0M 4s 2024-09-21T23:50:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56000K .......... .......... .......... .......... .......... 33% 24.6M 4s 2024-09-21T23:50:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56050K .......... .......... .......... .......... .......... 33% 47.6M 4s 2024-09-21T23:50:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56100K .......... .......... .......... .......... .......... 33% 25.2M 4s 2024-09-21T23:50:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56150K .......... .......... .......... .......... .......... 33% 34.5M 4s 2024-09-21T23:50:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56200K .......... .......... .......... .......... .......... 33% 25.0M 4s 2024-09-21T23:50:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56250K .......... .......... .......... .......... .......... 33% 62.0M 4s 2024-09-21T23:50:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56300K .......... .......... .......... .......... .......... 33% 22.2M 4s 2024-09-21T23:50:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56350K .......... .......... .......... .......... .......... 33% 24.2M 4s 2024-09-21T23:50:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56400K .......... .......... .......... .......... .......... 33% 23.1M 4s 2024-09-21T23:50:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56450K .......... .......... .......... .......... .......... 33% 37.8M 4s 2024-09-21T23:50:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56500K .......... .......... .......... .......... .......... 33% 24.5M 4s 2024-09-21T23:50:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56550K .......... .......... .......... .......... .......... 33% 46.3M 4s 2024-09-21T23:50:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56600K .......... .......... .......... .......... .......... 33% 24.9M 4s 2024-09-21T23:50:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56650K .......... .......... .......... .......... .......... 33% 37.6M 4s 2024-09-21T23:50:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56700K .......... .......... .......... .......... .......... 33% 25.1M 4s 2024-09-21T23:50:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56750K .......... .......... .......... .......... .......... 33% 23.9M 4s 2024-09-21T23:50:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56800K .......... .......... .......... .......... .......... 33% 46.6M 4s 2024-09-21T23:50:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56850K .......... .......... .......... .......... .......... 33% 23.0M 4s 2024-09-21T23:50:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56900K .......... .......... .......... .......... .......... 33% 39.7M 4s 2024-09-21T23:50:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 56950K .......... .......... .......... .......... .......... 33% 23.6M 4s 2024-09-21T23:50:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57000K .......... .......... .......... .......... .......... 33% 25.1M 4s 2024-09-21T23:50:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57050K .......... .......... .......... .......... .......... 33% 46.9M 4s 2024-09-21T23:50:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57100K .......... .......... .......... .......... .......... 33% 25.6M 4s 2024-09-21T23:50:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57150K .......... .......... .......... .......... .......... 33% 23.3M 4s 2024-09-21T23:50:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57200K .......... .......... .......... .......... .......... 33% 38.0M 4s 2024-09-21T23:50:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57250K .......... .......... .......... .......... .......... 33% 24.0M 4s 2024-09-21T23:50:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57300K .......... .......... .......... .......... .......... 34% 47.4M 4s 2024-09-21T23:50:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57350K .......... .......... .......... .......... .......... 34% 24.4M 4s 2024-09-21T23:50:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57400K .......... .......... .......... .......... .......... 34% 36.5M 4s 2024-09-21T23:50:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57450K .......... .......... .......... .......... .......... 34% 23.3M 4s 2024-09-21T23:50:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57500K .......... .......... .......... .......... .......... 34% 25.2M 4s 2024-09-21T23:50:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57550K .......... .......... .......... .......... .......... 34% 24.4M 4s 2024-09-21T23:50:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57600K .......... .......... .......... .......... .......... 34% 46.9M 4s 2024-09-21T23:50:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57650K .......... .......... .......... .......... .......... 34% 24.8M 4s 2024-09-21T23:50:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57700K .......... .......... .......... .......... .......... 34% 38.8M 4s 2024-09-21T23:50:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57750K .......... .......... .......... .......... .......... 34% 23.1M 4s 2024-09-21T23:50:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57800K .......... .......... .......... .......... .......... 34% 46.9M 4s 2024-09-21T23:50:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57850K .......... .......... .......... .......... .......... 34% 26.8M 4s 2024-09-21T23:50:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57900K .......... .......... .......... .......... .......... 34% 31.9M 4s 2024-09-21T23:50:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 57950K .......... .......... .......... .......... .......... 34% 22.9M 4s 2024-09-21T23:50:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58000K .......... .......... .......... .......... .......... 34% 24.7M 4s 2024-09-21T23:50:37.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58050K .......... .......... .......... .......... .......... 34% 24.9M 4s 2024-09-21T23:50:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58100K .......... .......... .......... .......... .......... 34% 47.6M 4s 2024-09-21T23:50:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58150K .......... .......... .......... .......... .......... 34% 23.6M 4s 2024-09-21T23:50:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58200K .......... .......... .......... .......... .......... 34% 37.7M 4s 2024-09-21T23:50:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58250K .......... .......... .......... .......... .......... 34% 24.7M 4s 2024-09-21T23:50:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58300K .......... .......... .......... .......... .......... 34% 52.0M 4s 2024-09-21T23:50:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58350K .......... .......... .......... .......... .......... 34% 23.6M 4s 2024-09-21T23:50:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58400K .......... .......... .......... .......... .......... 34% 24.2M 4s 2024-09-21T23:50:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58450K .......... .......... .......... .......... .......... 34% 34.6M 4s 2024-09-21T23:50:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58500K .......... .......... .......... .......... .......... 34% 26.1M 4s 2024-09-21T23:50:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58550K .......... .......... .......... .......... .......... 34% 21.0M 4s 2024-09-21T23:50:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58600K .......... .......... .......... .......... .......... 34% 72.6M 4s 2024-09-21T23:50:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58650K .......... .......... .......... .......... .......... 34% 32.9M 4s 2024-09-21T23:50:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58700K .......... .......... .......... .......... .......... 34% 24.6M 4s 2024-09-21T23:50:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58750K .......... .......... .......... .......... .......... 34% 22.9M 4s 2024-09-21T23:50:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58800K .......... .......... .......... .......... .......... 34% 25.2M 4s 2024-09-21T23:50:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58850K .......... .......... .......... .......... .......... 34% 42.1M 4s 2024-09-21T23:50:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58900K .......... .......... .......... .......... .......... 34% 25.9M 4s 2024-09-21T23:50:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 58950K .......... .......... .......... .......... .......... 35% 16.8M 4s 2024-09-21T23:50:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59000K .......... .......... .......... .......... .......... 35% 66.7M 4s 2024-09-21T23:50:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59050K .......... .......... .......... .......... .......... 35% 64.8M 4s 2024-09-21T23:50:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59100K .......... .......... .......... .......... .......... 35% 25.9M 4s 2024-09-21T23:50:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59150K .......... .......... .......... .......... .......... 35% 24.4M 4s 2024-09-21T23:50:37.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59200K .......... .......... .......... .......... .......... 35% 34.1M 4s 2024-09-21T23:50:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59250K .......... .......... .......... .......... .......... 35% 23.1M 4s 2024-09-21T23:50:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59300K .......... .......... .......... .......... .......... 35% 27.0M 4s 2024-09-21T23:50:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59350K .......... .......... .......... .......... .......... 35% 43.0M 4s 2024-09-21T23:50:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59400K .......... .......... .......... .......... .......... 35% 24.2M 4s 2024-09-21T23:50:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59450K .......... .......... .......... .......... .......... 35% 35.5M 4s 2024-09-21T23:50:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59500K .......... .......... .......... .......... .......... 35% 25.8M 4s 2024-09-21T23:50:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59550K .......... .......... .......... .......... .......... 35% 23.7M 4s 2024-09-21T23:50:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59600K .......... .......... .......... .......... .......... 35% 52.0M 4s 2024-09-21T23:50:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59650K .......... .......... .......... .......... .......... 35% 24.4M 4s 2024-09-21T23:50:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59700K .......... .......... .......... .......... .......... 35% 33.3M 4s 2024-09-21T23:50:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59750K .......... .......... .......... .......... .......... 35% 24.7M 4s 2024-09-21T23:50:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59800K .......... .......... .......... .......... .......... 35% 24.3M 4s 2024-09-21T23:50:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59850K .......... .......... .......... .......... .......... 35% 48.3M 4s 2024-09-21T23:50:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59900K .......... .......... .......... .......... .......... 35% 24.8M 4s 2024-09-21T23:50:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 59950K .......... .......... .......... .......... .......... 35% 24.2M 4s 2024-09-21T23:50:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60000K .......... .......... .......... .......... .......... 35% 36.2M 4s 2024-09-21T23:50:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60050K .......... .......... .......... .......... .......... 35% 24.9M 4s 2024-09-21T23:50:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60100K .......... .......... .......... .......... .......... 35% 50.3M 4s 2024-09-21T23:50:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60150K .......... .......... .......... .......... .......... 35% 24.9M 4s 2024-09-21T23:50:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60200K .......... .......... .......... .......... .......... 35% 24.2M 4s 2024-09-21T23:50:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60250K .......... .......... .......... .......... .......... 35% 38.4M 4s 2024-09-21T23:50:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60300K .......... .......... .......... .......... .......... 35% 23.5M 4s 2024-09-21T23:50:37.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60350K .......... .......... .......... .......... .......... 35% 30.1M 4s 2024-09-21T23:50:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60400K .......... .......... .......... .......... .......... 35% 35.2M 4s 2024-09-21T23:50:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60450K .......... .......... .......... .......... .......... 35% 23.2M 4s 2024-09-21T23:50:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60500K .......... .......... .......... .......... .......... 35% 34.8M 4s 2024-09-21T23:50:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60550K .......... .......... .......... .......... .......... 35% 28.8M 4s 2024-09-21T23:50:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60600K .......... .......... .......... .......... .......... 35% 34.5M 4s 2024-09-21T23:50:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60650K .......... .......... .......... .......... .......... 36% 25.7M 4s 2024-09-21T23:50:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60700K .......... .......... .......... .......... .......... 36% 28.8M 4s 2024-09-21T23:50:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60750K .......... .......... .......... .......... .......... 36% 12.0M 4s 2024-09-21T23:50:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60800K .......... .......... .......... .......... .......... 36% 70.6M 4s 2024-09-21T23:50:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60850K .......... .......... .......... .......... .......... 36% 78.7M 4s 2024-09-21T23:50:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60900K .......... .......... .......... .......... .......... 36% 43.0M 4s 2024-09-21T23:50:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 60950K .......... .......... .......... .......... .......... 36% 23.4M 4s 2024-09-21T23:50:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61000K .......... .......... .......... .......... .......... 36% 33.1M 4s 2024-09-21T23:50:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61050K .......... .......... .......... .......... .......... 36% 24.9M 4s 2024-09-21T23:50:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61100K .......... .......... .......... .......... .......... 36% 47.1M 4s 2024-09-21T23:50:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61150K .......... .......... .......... .......... .......... 36% 24.0M 4s 2024-09-21T23:50:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61200K .......... .......... .......... .......... .......... 36% 22.9M 4s 2024-09-21T23:50:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61250K .......... .......... .......... .......... .......... 36% 40.2M 4s 2024-09-21T23:50:37.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61300K .......... .......... .......... .......... .......... 36% 26.5M 4s 2024-09-21T23:50:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61350K .......... .......... .......... .......... .......... 36% 21.2M 4s 2024-09-21T23:50:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61400K .......... .......... .......... .......... .......... 36% 62.0M 4s 2024-09-21T23:50:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61450K .......... .......... .......... .......... .......... 36% 35.1M 4s 2024-09-21T23:50:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61500K .......... .......... .......... .......... .......... 36% 24.2M 4s 2024-09-21T23:50:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61550K .......... .......... .......... .......... .......... 36% 24.2M 4s 2024-09-21T23:50:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61600K .......... .......... .......... .......... .......... 36% 24.8M 4s 2024-09-21T23:50:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61650K .......... .......... .......... .......... .......... 36% 27.9M 4s 2024-09-21T23:50:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61700K .......... .......... .......... .......... .......... 36% 40.0M 4s 2024-09-21T23:50:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61750K .......... .......... .......... .......... .......... 36% 26.8M 4s 2024-09-21T23:50:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61800K .......... .......... .......... .......... .......... 36% 23.4M 4s 2024-09-21T23:50:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61850K .......... .......... .......... .......... .......... 36% 74.9M 4s 2024-09-21T23:50:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61900K .......... .......... .......... .......... .......... 36% 27.1M 4s 2024-09-21T23:50:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 61950K .......... .......... .......... .......... .......... 36% 24.3M 4s 2024-09-21T23:50:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62000K .......... .......... .......... .......... .......... 36% 23.3M 4s 2024-09-21T23:50:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62050K .......... .......... .......... .......... .......... 36% 33.5M 4s 2024-09-21T23:50:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62100K .......... .......... .......... .......... .......... 36% 26.5M 4s 2024-09-21T23:50:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62150K .......... .......... .......... .......... .......... 36% 40.9M 4s 2024-09-21T23:50:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62200K .......... .......... .......... .......... .......... 36% 27.6M 4s 2024-09-21T23:50:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62250K .......... .......... .......... .......... .......... 36% 34.6M 4s 2024-09-21T23:50:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62300K .......... .......... .......... .......... .......... 36% 27.1M 4s 2024-09-21T23:50:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62350K .......... .......... .......... .......... .......... 37% 22.7M 4s 2024-09-21T23:50:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62400K .......... .......... .......... .......... .......... 37% 42.7M 4s 2024-09-21T23:50:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62450K .......... .......... .......... .......... .......... 37% 26.1M 4s 2024-09-21T23:50:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62500K .......... .......... .......... .......... .......... 37% 35.5M 4s 2024-09-21T23:50:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62550K .......... .......... .......... .......... .......... 37% 23.9M 4s 2024-09-21T23:50:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62600K .......... .......... .......... .......... .......... 37% 25.5M 4s 2024-09-21T23:50:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62650K .......... .......... .......... .......... .......... 37% 46.0M 4s 2024-09-21T23:50:37.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62700K .......... .......... .......... .......... .......... 37% 22.9M 4s 2024-09-21T23:50:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62750K .......... .......... .......... .......... .......... 37% 25.7M 4s 2024-09-21T23:50:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62800K .......... .......... .......... .......... .......... 37% 36.6M 4s 2024-09-21T23:50:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62850K .......... .......... .......... .......... .......... 37% 25.1M 4s 2024-09-21T23:50:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62900K .......... .......... .......... .......... .......... 37% 46.9M 4s 2024-09-21T23:50:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 62950K .......... .......... .......... .......... .......... 37% 24.1M 4s 2024-09-21T23:50:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63000K .......... .......... .......... .......... .......... 37% 39.3M 4s 2024-09-21T23:50:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63050K .......... .......... .......... .......... .......... 37% 23.6M 4s 2024-09-21T23:50:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63100K .......... .......... .......... .......... .......... 37% 24.0M 4s 2024-09-21T23:50:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63150K .......... .......... .......... .......... .......... 37% 23.0M 4s 2024-09-21T23:50:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63200K .......... .......... .......... .......... .......... 37% 52.1M 4s 2024-09-21T23:50:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63250K .......... .......... .......... .......... .......... 37% 24.6M 4s 2024-09-21T23:50:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63300K .......... .......... .......... .......... .......... 37% 34.4M 4s 2024-09-21T23:50:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63350K .......... .......... .......... .......... .......... 37% 24.5M 4s 2024-09-21T23:50:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63400K .......... .......... .......... .......... .......... 37% 56.4M 4s 2024-09-21T23:50:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63450K .......... .......... .......... .......... .......... 37% 34.1M 4s 2024-09-21T23:50:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63500K .......... .......... .......... .......... .......... 37% 23.1M 4s 2024-09-21T23:50:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63550K .......... .......... .......... .......... .......... 37% 24.2M 4s 2024-09-21T23:50:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63600K .......... .......... .......... .......... .......... 37% 25.2M 4s 2024-09-21T23:50:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63650K .......... .......... .......... .......... .......... 37% 23.4M 4s 2024-09-21T23:50:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63700K .......... .......... .......... .......... .......... 37% 89.5M 4s 2024-09-21T23:50:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63750K .......... .......... .......... .......... .......... 37% 26.3M 4s 2024-09-21T23:50:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63800K .......... .......... .......... .......... .......... 37% 25.7M 4s 2024-09-21T23:50:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63850K .......... .......... .......... .......... .......... 37% 22.8M 4s 2024-09-21T23:50:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63900K .......... .......... .......... .......... .......... 37% 71.8M 4s 2024-09-21T23:50:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 63950K .......... .......... .......... .......... .......... 37% 22.5M 4s 2024-09-21T23:50:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64000K .......... .......... .......... .......... .......... 37% 30.3M 4s 2024-09-21T23:50:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64050K .......... .......... .......... .......... .......... 38% 28.8M 4s 2024-09-21T23:50:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64100K .......... .......... .......... .......... .......... 38% 24.8M 4s 2024-09-21T23:50:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64150K .......... .......... .......... .......... .......... 38% 45.4M 4s 2024-09-21T23:50:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64200K .......... .......... .......... .......... .......... 38% 24.6M 4s 2024-09-21T23:50:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64250K .......... .......... .......... .......... .......... 38% 35.0M 4s 2024-09-21T23:50:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64300K .......... .......... .......... .......... .......... 38% 22.4M 4s 2024-09-21T23:50:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64350K .......... .......... .......... .......... .......... 38% 23.8M 4s 2024-09-21T23:50:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64400K .......... .......... .......... .......... .......... 38% 25.5M 4s 2024-09-21T23:50:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64450K .......... .......... .......... .......... .......... 38% 55.7M 4s 2024-09-21T23:50:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64500K .......... .......... .......... .......... .......... 38% 23.2M 4s 2024-09-21T23:50:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64550K .......... .......... .......... .......... .......... 38% 36.0M 4s 2024-09-21T23:50:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64600K .......... .......... .......... .......... .......... 38% 24.4M 4s 2024-09-21T23:50:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64650K .......... .......... .......... .......... .......... 38% 49.7M 4s 2024-09-21T23:50:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64700K .......... .......... .......... .......... .......... 38% 24.7M 4s 2024-09-21T23:50:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64750K .......... .......... .......... .......... .......... 38% 23.8M 4s 2024-09-21T23:50:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64800K .......... .......... .......... .......... .......... 38% 34.6M 4s 2024-09-21T23:50:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64850K .......... .......... .......... .......... .......... 38% 25.9M 4s 2024-09-21T23:50:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64900K .......... .......... .......... .......... .......... 38% 23.1M 4s 2024-09-21T23:50:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 64950K .......... .......... .......... .......... .......... 38% 51.1M 4s 2024-09-21T23:50:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65000K .......... .......... .......... .......... .......... 38% 23.6M 4s 2024-09-21T23:50:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65050K .......... .......... .......... .......... .......... 38% 40.1M 4s 2024-09-21T23:50:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65100K .......... .......... .......... .......... .......... 38% 24.4M 4s 2024-09-21T23:50:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65150K .......... .......... .......... .......... .......... 38% 23.8M 4s 2024-09-21T23:50:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65200K .......... .......... .......... .......... .......... 38% 46.1M 4s 2024-09-21T23:50:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65250K .......... .......... .......... .......... .......... 38% 25.3M 4s 2024-09-21T23:50:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65300K .......... .......... .......... .......... .......... 38% 35.3M 4s 2024-09-21T23:50:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65350K .......... .......... .......... .......... .......... 38% 23.5M 4s 2024-09-21T23:50:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65400K .......... .......... .......... .......... .......... 38% 24.8M 4s 2024-09-21T23:50:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65450K .......... .......... .......... .......... .......... 38% 52.2M 4s 2024-09-21T23:50:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65500K .......... .......... .......... .......... .......... 38% 24.3M 4s 2024-09-21T23:50:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65550K .......... .......... .......... .......... .......... 38% 23.2M 4s 2024-09-21T23:50:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65600K .......... .......... .......... .......... .......... 38% 37.2M 4s 2024-09-21T23:50:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65650K .......... .......... .......... .......... .......... 38% 24.7M 4s 2024-09-21T23:50:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65700K .......... .......... .......... .......... .......... 39% 47.1M 4s 2024-09-21T23:50:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65750K .......... .......... .......... .......... .......... 39% 24.5M 4s 2024-09-21T23:50:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65800K .......... .......... .......... .......... .......... 39% 36.3M 4s 2024-09-21T23:50:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65850K .......... .......... .......... .......... .......... 39% 24.4M 4s 2024-09-21T23:50:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65900K .......... .......... .......... .......... .......... 39% 23.0M 4s 2024-09-21T23:50:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 65950K .......... .......... .......... .......... .......... 39% 24.2M 4s 2024-09-21T23:50:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66000K .......... .......... .......... .......... .......... 39% 56.5M 4s 2024-09-21T23:50:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66050K .......... .......... .......... .......... .......... 39% 23.9M 4s 2024-09-21T23:50:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66100K .......... .......... .......... .......... .......... 39% 34.8M 4s 2024-09-21T23:50:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66150K .......... .......... .......... .......... .......... 39% 24.9M 4s 2024-09-21T23:50:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66200K .......... .......... .......... .......... .......... 39% 51.0M 4s 2024-09-21T23:50:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66250K .......... .......... .......... .......... .......... 39% 24.7M 4s 2024-09-21T23:50:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66300K .......... .......... .......... .......... .......... 39% 35.1M 4s 2024-09-21T23:50:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66350K .......... .......... .......... .......... .......... 39% 23.6M 4s 2024-09-21T23:50:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66400K .......... .......... .......... .......... .......... 39% 29.0M 4s 2024-09-21T23:50:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66450K .......... .......... .......... .......... .......... 39% 20.0M 4s 2024-09-21T23:50:37.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66500K .......... .......... .......... .......... .......... 39% 70.4M 4s 2024-09-21T23:50:37.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66550K .......... .......... .......... .......... .......... 39% 31.5M 4s 2024-09-21T23:50:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66600K .......... .......... .......... .......... .......... 39% 24.6M 4s 2024-09-21T23:50:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66650K .......... .......... .......... .......... .......... 39% 25.1M 4s 2024-09-21T23:50:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66700K .......... .......... .......... .......... .......... 39% 42.7M 4s 2024-09-21T23:50:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66750K .......... .......... .......... .......... .......... 39% 24.4M 4s 2024-09-21T23:50:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66800K .......... .......... .......... .......... .......... 39% 24.7M 4s 2024-09-21T23:50:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66850K .......... .......... .......... .......... .......... 39% 37.9M 4s 2024-09-21T23:50:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66900K .......... .......... .......... .......... .......... 39% 24.3M 4s 2024-09-21T23:50:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 66950K .......... .......... .......... .......... .......... 39% 45.7M 4s 2024-09-21T23:50:37.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67000K .......... .......... .......... .......... .......... 39% 25.2M 4s 2024-09-21T23:50:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67050K .......... .......... .......... .......... .......... 39% 33.7M 4s 2024-09-21T23:50:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67100K .......... .......... .......... .......... .......... 39% 21.1M 4s 2024-09-21T23:50:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67150K .......... .......... .......... .......... .......... 39% 28.3M 4s 2024-09-21T23:50:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67200K .......... .......... .......... .......... .......... 39% 24.4M 4s 2024-09-21T23:50:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67250K .......... .......... .......... .......... .......... 39% 52.1M 4s 2024-09-21T23:50:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67300K .......... .......... .......... .......... .......... 39% 25.1M 4s 2024-09-21T23:50:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67350K .......... .......... .......... .......... .......... 39% 27.8M 4s 2024-09-21T23:50:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67400K .......... .......... .......... .......... .......... 40% 35.9M 4s 2024-09-21T23:50:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67450K .......... .......... .......... .......... .......... 40% 33.4M 4s 2024-09-21T23:50:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67500K .......... .......... .......... .......... .......... 40% 24.8M 4s 2024-09-21T23:50:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67550K .......... .......... .......... .......... .......... 40% 24.5M 4s 2024-09-21T23:50:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67600K .......... .......... .......... .......... .......... 40% 33.5M 4s 2024-09-21T23:50:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67650K .......... .......... .......... .......... .......... 40% 23.9M 4s 2024-09-21T23:50:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67700K .......... .......... .......... .......... .......... 40% 25.0M 4s 2024-09-21T23:50:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67750K .......... .......... .......... .......... .......... 40% 45.9M 4s 2024-09-21T23:50:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67800K .......... .......... .......... .......... .......... 40% 24.6M 4s 2024-09-21T23:50:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67850K .......... .......... .......... .......... .......... 40% 36.8M 4s 2024-09-21T23:50:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67900K .......... .......... .......... .......... .......... 40% 24.0M 4s 2024-09-21T23:50:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 67950K .......... .......... .......... .......... .......... 40% 24.9M 4s 2024-09-21T23:50:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68000K .......... .......... .......... .......... .......... 40% 46.7M 4s 2024-09-21T23:50:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68050K .......... .......... .......... .......... .......... 40% 23.9M 4s 2024-09-21T23:50:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68100K .......... .......... .......... .......... .......... 40% 34.1M 4s 2024-09-21T23:50:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68150K .......... .......... .......... .......... .......... 40% 26.9M 4s 2024-09-21T23:50:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68200K .......... .......... .......... .......... .......... 40% 12.7M 4s 2024-09-21T23:50:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68250K .......... .......... .......... .......... .......... 40% 80.0M 4s 2024-09-21T23:50:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68300K .......... .......... .......... .......... .......... 40% 76.2M 4s 2024-09-21T23:50:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68350K .......... .......... .......... .......... .......... 40% 23.8M 4s 2024-09-21T23:50:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68400K .......... .......... .......... .......... .......... 40% 39.1M 4s 2024-09-21T23:50:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68450K .......... .......... .......... .......... .......... 40% 24.8M 4s 2024-09-21T23:50:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68500K .......... .......... .......... .......... .......... 40% 47.2M 4s 2024-09-21T23:50:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68550K .......... .......... .......... .......... .......... 40% 10.2M 4s 2024-09-21T23:50:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68600K .......... .......... .......... .......... .......... 40% 107M 4s 2024-09-21T23:50:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68650K .......... .......... .......... .......... .......... 40% 90.2M 4s 2024-09-21T23:50:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68700K .......... .......... .......... .......... .......... 40% 78.3M 4s 2024-09-21T23:50:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68750K .......... .......... .......... .......... .......... 40% 20.0M 4s 2024-09-21T23:50:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68800K .......... .......... .......... .......... .......... 40% 36.8M 4s 2024-09-21T23:50:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68850K .......... .......... .......... .......... .......... 40% 29.1M 4s 2024-09-21T23:50:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68900K .......... .......... .......... .......... .......... 40% 25.1M 4s 2024-09-21T23:50:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 68950K .......... .......... .......... .......... .......... 40% 24.1M 4s 2024-09-21T23:50:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69000K .......... .......... .......... .......... .......... 40% 51.2M 4s 2024-09-21T23:50:37.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69050K .......... .......... .......... .......... .......... 40% 24.8M 4s 2024-09-21T23:50:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69100K .......... .......... .......... .......... .......... 41% 36.3M 4s 2024-09-21T23:50:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69150K .......... .......... .......... .......... .......... 41% 28.1M 4s 2024-09-21T23:50:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69200K .......... .......... .......... .......... .......... 41% 22.3M 4s 2024-09-21T23:50:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69250K .......... .......... .......... .......... .......... 41% 45.1M 4s 2024-09-21T23:50:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69300K .......... .......... .......... .......... .......... 41% 25.1M 4s 2024-09-21T23:50:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69350K .......... .......... .......... .......... .......... 41% 33.5M 3s 2024-09-21T23:50:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69400K .......... .......... .......... .......... .......... 41% 23.3M 3s 2024-09-21T23:50:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69450K .......... .......... .......... .......... .......... 41% 26.4M 3s 2024-09-21T23:50:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69500K .......... .......... .......... .......... .......... 41% 46.1M 3s 2024-09-21T23:50:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69550K .......... .......... .......... .......... .......... 41% 22.9M 3s 2024-09-21T23:50:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69600K .......... .......... .......... .......... .......... 41% 25.6M 3s 2024-09-21T23:50:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69650K .......... .......... .......... .......... .......... 41% 40.9M 3s 2024-09-21T23:50:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69700K .......... .......... .......... .......... .......... 41% 21.3M 3s 2024-09-21T23:50:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69750K .......... .......... .......... .......... .......... 41% 49.0M 3s 2024-09-21T23:50:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69800K .......... .......... .......... .......... .......... 41% 25.1M 3s 2024-09-21T23:50:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69850K .......... .......... .......... .......... .......... 41% 37.6M 3s 2024-09-21T23:50:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69900K .......... .......... .......... .......... .......... 41% 23.9M 3s 2024-09-21T23:50:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 69950K .......... .......... .......... .......... .......... 41% 23.8M 3s 2024-09-21T23:50:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70000K .......... .......... .......... .......... .......... 41% 25.0M 3s 2024-09-21T23:50:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70050K .......... .......... .......... .......... .......... 41% 46.6M 3s 2024-09-21T23:50:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70100K .......... .......... .......... .......... .......... 41% 24.8M 3s 2024-09-21T23:50:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70150K .......... .......... .......... .......... .......... 41% 36.1M 3s 2024-09-21T23:50:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70200K .......... .......... .......... .......... .......... 41% 26.1M 3s 2024-09-21T23:50:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70250K .......... .......... .......... .......... .......... 41% 44.2M 3s 2024-09-21T23:50:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70300K .......... .......... .......... .......... .......... 41% 24.7M 3s 2024-09-21T23:50:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70350K .......... .......... .......... .......... .......... 41% 23.9M 3s 2024-09-21T23:50:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70400K .......... .......... .......... .......... .......... 41% 38.7M 3s 2024-09-21T23:50:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70450K .......... .......... .......... .......... .......... 41% 22.3M 3s 2024-09-21T23:50:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70500K .......... .......... .......... .......... .......... 41% 24.5M 3s 2024-09-21T23:50:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70550K .......... .......... .......... .......... .......... 41% 49.7M 3s 2024-09-21T23:50:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70600K .......... .......... .......... .......... .......... 41% 24.0M 3s 2024-09-21T23:50:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70650K .......... .......... .......... .......... .......... 41% 37.4M 3s 2024-09-21T23:50:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70700K .......... .......... .......... .......... .......... 41% 32.7M 3s 2024-09-21T23:50:37.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70750K .......... .......... .......... .......... .......... 42% 19.9M 3s 2024-09-21T23:50:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70800K .......... .......... .......... .......... .......... 42% 45.6M 3s 2024-09-21T23:50:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70850K .......... .......... .......... .......... .......... 42% 25.0M 3s 2024-09-21T23:50:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70900K .......... .......... .......... .......... .......... 42% 36.6M 3s 2024-09-21T23:50:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 70950K .......... .......... .......... .......... .......... 42% 24.4M 3s 2024-09-21T23:50:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71000K .......... .......... .......... .......... .......... 42% 22.9M 3s 2024-09-21T23:50:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71050K .......... .......... .......... .......... .......... 42% 52.3M 3s 2024-09-21T23:50:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71100K .......... .......... .......... .......... .......... 42% 35.7M 3s 2024-09-21T23:50:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71150K .......... .......... .......... .......... .......... 42% 18.4M 3s 2024-09-21T23:50:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71200K .......... .......... .......... .......... .......... 42% 36.5M 3s 2024-09-21T23:50:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71250K .......... .......... .......... .......... .......... 42% 24.8M 3s 2024-09-21T23:50:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71300K .......... .......... .......... .......... .......... 42% 47.8M 3s 2024-09-21T23:50:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71350K .......... .......... .......... .......... .......... 42% 24.2M 3s 2024-09-21T23:50:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71400K .......... .......... .......... .......... .......... 42% 35.7M 3s 2024-09-21T23:50:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71450K .......... .......... .......... .......... .......... 42% 24.4M 3s 2024-09-21T23:50:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71500K .......... .......... .......... .......... .......... 42% 53.7M 3s 2024-09-21T23:50:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71550K .......... .......... .......... .......... .......... 42% 15.4M 3s 2024-09-21T23:50:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71600K .......... .......... .......... .......... .......... 42% 56.1M 3s 2024-09-21T23:50:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71650K .......... .......... .......... .......... .......... 42% 31.4M 3s 2024-09-21T23:50:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71700K .......... .......... .......... .......... .......... 42% 26.1M 3s 2024-09-21T23:50:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71750K .......... .......... .......... .......... .......... 42% 24.1M 3s 2024-09-21T23:50:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71800K .......... .......... .......... .......... .......... 42% 49.6M 3s 2024-09-21T23:50:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71850K .......... .......... .......... .......... .......... 42% 25.0M 3s 2024-09-21T23:50:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71900K .......... .......... .......... .......... .......... 42% 36.4M 3s 2024-09-21T23:50:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 71950K .......... .......... .......... .......... .......... 42% 23.8M 3s 2024-09-21T23:50:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72000K .......... .......... .......... .......... .......... 42% 25.0M 3s 2024-09-21T23:50:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72050K .......... .......... .......... .......... .......... 42% 47.9M 3s 2024-09-21T23:50:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72100K .......... .......... .......... .......... .......... 42% 25.3M 3s 2024-09-21T23:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72150K .......... .......... .......... .......... .......... 42% 31.4M 3s 2024-09-21T23:50:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72200K .......... .......... .......... .......... .......... 42% 24.6M 3s 2024-09-21T23:50:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72250K .......... .......... .......... .......... .......... 42% 24.9M 3s 2024-09-21T23:50:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72300K .......... .......... .......... .......... .......... 42% 50.6M 3s 2024-09-21T23:50:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72350K .......... .......... .......... .......... .......... 42% 23.8M 3s 2024-09-21T23:50:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72400K .......... .......... .......... .......... .......... 42% 24.9M 3s 2024-09-21T23:50:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72450K .......... .......... .......... .......... .......... 43% 35.6M 3s 2024-09-21T23:50:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72500K .......... .......... .......... .......... .......... 43% 24.8M 3s 2024-09-21T23:50:37.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72550K .......... .......... .......... .......... .......... 43% 46.0M 3s 2024-09-21T23:50:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72600K .......... .......... .......... .......... .......... 43% 26.0M 3s 2024-09-21T23:50:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72650K .......... .......... .......... .......... .......... 43% 34.5M 3s 2024-09-21T23:50:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72700K .......... .......... .......... .......... .......... 43% 24.1M 3s 2024-09-21T23:50:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72750K .......... .......... .......... .......... .......... 43% 24.5M 3s 2024-09-21T23:50:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72800K .......... .......... .......... .......... .......... 43% 24.0M 3s 2024-09-21T23:50:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72850K .......... .......... .......... .......... .......... 43% 48.9M 3s 2024-09-21T23:50:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72900K .......... .......... .......... .......... .......... 43% 25.7M 3s 2024-09-21T23:50:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 72950K .......... .......... .......... .......... .......... 43% 33.0M 3s 2024-09-21T23:50:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73000K .......... .......... .......... .......... .......... 43% 24.4M 3s 2024-09-21T23:50:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73050K .......... .......... .......... .......... .......... 43% 47.9M 3s 2024-09-21T23:50:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73100K .......... .......... .......... .......... .......... 43% 27.8M 3s 2024-09-21T23:50:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73150K .......... .......... .......... .......... .......... 43% 22.7M 3s 2024-09-21T23:50:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73200K .......... .......... .......... .......... .......... 43% 35.0M 3s 2024-09-21T23:50:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73250K .......... .......... .......... .......... .......... 43% 24.2M 3s 2024-09-21T23:50:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73300K .......... .......... .......... .......... .......... 43% 24.7M 3s 2024-09-21T23:50:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73350K .......... .......... .......... .......... .......... 43% 48.0M 3s 2024-09-21T23:50:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73400K .......... .......... .......... .......... .......... 43% 23.1M 3s 2024-09-21T23:50:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73450K .......... .......... .......... .......... .......... 43% 38.3M 3s 2024-09-21T23:50:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73500K .......... .......... .......... .......... .......... 43% 26.8M 3s 2024-09-21T23:50:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73550K .......... .......... .......... .......... .......... 43% 22.8M 3s 2024-09-21T23:50:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73600K .......... .......... .......... .......... .......... 43% 49.2M 3s 2024-09-21T23:50:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73650K .......... .......... .......... .......... .......... 43% 24.9M 3s 2024-09-21T23:50:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73700K .......... .......... .......... .......... .......... 43% 36.6M 3s 2024-09-21T23:50:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73750K .......... .......... .......... .......... .......... 43% 23.4M 3s 2024-09-21T23:50:38.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73800K .......... .......... .......... .......... .......... 43% 48.3M 3s 2024-09-21T23:50:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73850K .......... .......... .......... .......... .......... 43% 25.3M 3s 2024-09-21T23:50:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73900K .......... .......... .......... .......... .......... 43% 32.0M 3s 2024-09-21T23:50:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 73950K .......... .......... .......... .......... .......... 43% 18.4M 3s 2024-09-21T23:50:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74000K .......... .......... .......... .......... .......... 43% 38.1M 3s 2024-09-21T23:50:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74050K .......... .......... .......... .......... .......... 43% 24.6M 3s 2024-09-21T23:50:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74100K .......... .......... .......... .......... .......... 43% 45.5M 3s 2024-09-21T23:50:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74150K .......... .......... .......... .......... .......... 44% 25.0M 3s 2024-09-21T23:50:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74200K .......... .......... .......... .......... .......... 44% 37.2M 3s 2024-09-21T23:50:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74250K .......... .......... .......... .......... .......... 44% 23.4M 3s 2024-09-21T23:50:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74300K .......... .......... .......... .......... .......... 44% 48.6M 3s 2024-09-21T23:50:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74350K .......... .......... .......... .......... .......... 44% 23.4M 3s 2024-09-21T23:50:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74400K .......... .......... .......... .......... .......... 44% 25.1M 3s 2024-09-21T23:50:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74450K .......... .......... .......... .......... .......... 44% 33.1M 3s 2024-09-21T23:50:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74500K .......... .......... .......... .......... .......... 44% 24.7M 3s 2024-09-21T23:50:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74550K .......... .......... .......... .......... .......... 44% 23.8M 3s 2024-09-21T23:50:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74600K .......... .......... .......... .......... .......... 44% 58.7M 3s 2024-09-21T23:50:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74650K .......... .......... .......... .......... .......... 44% 30.8M 3s 2024-09-21T23:50:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74700K .......... .......... .......... .......... .......... 44% 19.3M 3s 2024-09-21T23:50:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74750K .......... .......... .......... .......... .......... 44% 42.2M 3s 2024-09-21T23:50:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74800K .......... .......... .......... .......... .......... 44% 23.0M 3s 2024-09-21T23:50:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74850K .......... .......... .......... .......... .......... 44% 48.5M 3s 2024-09-21T23:50:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74900K .......... .......... .......... .......... .......... 44% 25.0M 3s 2024-09-21T23:50:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 74950K .......... .......... .......... .......... .......... 44% 34.9M 3s 2024-09-21T23:50:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75000K .......... .......... .......... .......... .......... 44% 22.8M 3s 2024-09-21T23:50:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75050K .......... .......... .......... .......... .......... 44% 25.7M 3s 2024-09-21T23:50:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75100K .......... .......... .......... .......... .......... 44% 53.0M 3s 2024-09-21T23:50:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75150K .......... .......... .......... .......... .......... 44% 23.0M 3s 2024-09-21T23:50:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75200K .......... .......... .......... .......... .......... 44% 24.9M 3s 2024-09-21T23:50:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75250K .......... .......... .......... .......... .......... 44% 34.2M 3s 2024-09-21T23:50:38.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75300K .......... .......... .......... .......... .......... 44% 26.4M 3s 2024-09-21T23:50:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75350K .......... .......... .......... .......... .......... 44% 47.3M 3s 2024-09-21T23:50:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75400K .......... .......... .......... .......... .......... 44% 24.0M 3s 2024-09-21T23:50:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75450K .......... .......... .......... .......... .......... 44% 36.2M 3s 2024-09-21T23:50:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75500K .......... .......... .......... .......... .......... 44% 24.7M 3s 2024-09-21T23:50:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75550K .......... .......... .......... .......... .......... 44% 22.2M 3s 2024-09-21T23:50:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75600K .......... .......... .......... .......... .......... 44% 25.1M 3s 2024-09-21T23:50:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75650K .......... .......... .......... .......... .......... 44% 54.3M 3s 2024-09-21T23:50:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75700K .......... .......... .......... .......... .......... 44% 24.4M 3s 2024-09-21T23:50:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75750K .......... .......... .......... .......... .......... 44% 33.9M 3s 2024-09-21T23:50:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75800K .......... .......... .......... .......... .......... 44% 25.0M 3s 2024-09-21T23:50:38.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75850K .......... .......... .......... .......... .......... 45% 49.9M 3s 2024-09-21T23:50:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75900K .......... .......... .......... .......... .......... 45% 24.2M 3s 2024-09-21T23:50:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 75950K .......... .......... .......... .......... .......... 45% 24.1M 3s 2024-09-21T23:50:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76000K .......... .......... .......... .......... .......... 45% 36.4M 3s 2024-09-21T23:50:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76050K .......... .......... .......... .......... .......... 45% 19.8M 3s 2024-09-21T23:50:38.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76100K .......... .......... .......... .......... .......... 45% 31.7M 3s 2024-09-21T23:50:38.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76150K .......... .......... .......... .......... .......... 45% 46.3M 3s 2024-09-21T23:50:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76200K .......... .......... .......... .......... .......... 45% 34.9M 3s 2024-09-21T23:50:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76250K .......... .......... .......... .......... .......... 45% 23.6M 3s 2024-09-21T23:50:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76300K .......... .......... .......... .......... .......... 45% 26.1M 3s 2024-09-21T23:50:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76350K .......... .......... .......... .......... .......... 45% 24.8M 3s 2024-09-21T23:50:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76400K .......... .......... .......... .......... .......... 45% 40.2M 3s 2024-09-21T23:50:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76450K .......... .......... .......... .......... .......... 45% 27.0M 3s 2024-09-21T23:50:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76500K .......... .......... .......... .......... .......... 45% 35.2M 3s 2024-09-21T23:50:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76550K .......... .......... .......... .......... .......... 45% 24.7M 3s 2024-09-21T23:50:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76600K .......... .......... .......... .......... .......... 45% 48.6M 3s 2024-09-21T23:50:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76650K .......... .......... .......... .......... .......... 45% 24.5M 3s 2024-09-21T23:50:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76700K .......... .......... .......... .......... .......... 45% 34.0M 3s 2024-09-21T23:50:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76750K .......... .......... .......... .......... .......... 45% 24.5M 3s 2024-09-21T23:50:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76800K .......... .......... .......... .......... .......... 45% 25.5M 3s 2024-09-21T23:50:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76850K .......... .......... .......... .......... .......... 45% 22.9M 3s 2024-09-21T23:50:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76900K .......... .......... .......... .......... .......... 45% 48.5M 3s 2024-09-21T23:50:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 76950K .......... .......... .......... .......... .......... 45% 24.8M 3s 2024-09-21T23:50:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77000K .......... .......... .......... .......... .......... 45% 34.6M 3s 2024-09-21T23:50:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77050K .......... .......... .......... .......... .......... 45% 25.4M 3s 2024-09-21T23:50:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77100K .......... .......... .......... .......... .......... 45% 45.0M 3s 2024-09-21T23:50:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77150K .......... .......... .......... .......... .......... 45% 24.3M 3s 2024-09-21T23:50:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77200K .......... .......... .......... .......... .......... 45% 24.7M 3s 2024-09-21T23:50:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77250K .......... .......... .......... .......... .......... 45% 33.6M 3s 2024-09-21T23:50:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77300K .......... .......... .......... .......... .......... 45% 23.6M 3s 2024-09-21T23:50:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77350K .......... .......... .......... .......... .......... 45% 26.4M 3s 2024-09-21T23:50:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77400K .......... .......... .......... .......... .......... 45% 45.9M 3s 2024-09-21T23:50:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77450K .......... .......... .......... .......... .......... 45% 25.2M 3s 2024-09-21T23:50:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77500K .......... .......... .......... .......... .......... 46% 43.4M 3s 2024-09-21T23:50:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77550K .......... .......... .......... .......... .......... 46% 25.5M 3s 2024-09-21T23:50:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77600K .......... .......... .......... .......... .......... 46% 21.4M 3s 2024-09-21T23:50:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77650K .......... .......... .......... .......... .......... 46% 42.6M 3s 2024-09-21T23:50:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77700K .......... .......... .......... .......... .......... 46% 26.9M 3s 2024-09-21T23:50:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77750K .......... .......... .......... .......... .......... 46% 32.5M 3s 2024-09-21T23:50:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77800K .......... .......... .......... .......... .......... 46% 23.2M 3s 2024-09-21T23:50:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77850K .......... .......... .......... .......... .......... 46% 26.8M 3s 2024-09-21T23:50:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77900K .......... .......... .......... .......... .......... 46% 52.1M 3s 2024-09-21T23:50:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 77950K .......... .......... .......... .......... .......... 46% 23.5M 3s 2024-09-21T23:50:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78000K .......... .......... .......... .......... .......... 46% 24.5M 3s 2024-09-21T23:50:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78050K .......... .......... .......... .......... .......... 46% 33.6M 3s 2024-09-21T23:50:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78100K .......... .......... .......... .......... .......... 46% 27.3M 3s 2024-09-21T23:50:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78150K .......... .......... .......... .......... .......... 46% 43.7M 3s 2024-09-21T23:50:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78200K .......... .......... .......... .......... .......... 46% 23.2M 3s 2024-09-21T23:50:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78250K .......... .......... .......... .......... .......... 46% 39.2M 3s 2024-09-21T23:50:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78300K .......... .......... .......... .......... .......... 46% 23.8M 3s 2024-09-21T23:50:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78350K .......... .......... .......... .......... .......... 46% 22.7M 3s 2024-09-21T23:50:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78400K .......... .......... .......... .......... .......... 46% 25.7M 3s 2024-09-21T23:50:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78450K .......... .......... .......... .......... .......... 46% 53.7M 3s 2024-09-21T23:50:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78500K .......... .......... .......... .......... .......... 46% 24.2M 3s 2024-09-21T23:50:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78550K .......... .......... .......... .......... .......... 46% 34.1M 3s 2024-09-21T23:50:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78600K .......... .......... .......... .......... .......... 46% 24.1M 3s 2024-09-21T23:50:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78650K .......... .......... .......... .......... .......... 46% 50.1M 3s 2024-09-21T23:50:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78700K .......... .......... .......... .......... .......... 46% 24.9M 3s 2024-09-21T23:50:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78750K .......... .......... .......... .......... .......... 46% 24.2M 3s 2024-09-21T23:50:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78800K .......... .......... .......... .......... .......... 46% 33.2M 3s 2024-09-21T23:50:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78850K .......... .......... .......... .......... .......... 46% 25.7M 3s 2024-09-21T23:50:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78900K .......... .......... .......... .......... .......... 46% 23.6M 3s 2024-09-21T23:50:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 78950K .......... .......... .......... .......... .......... 46% 54.5M 3s 2024-09-21T23:50:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79000K .......... .......... .......... .......... .......... 46% 32.4M 3s 2024-09-21T23:50:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79050K .......... .......... .......... .......... .......... 46% 25.5M 3s 2024-09-21T23:50:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79100K .......... .......... .......... .......... .......... 46% 23.9M 3s 2024-09-21T23:50:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79150K .......... .......... .......... .......... .......... 46% 24.8M 3s 2024-09-21T23:50:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79200K .......... .......... .......... .......... .......... 47% 50.2M 3s 2024-09-21T23:50:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79250K .......... .......... .......... .......... .......... 47% 24.5M 3s 2024-09-21T23:50:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79300K .......... .......... .......... .......... .......... 47% 35.9M 3s 2024-09-21T23:50:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79350K .......... .......... .......... .......... .......... 47% 24.1M 3s 2024-09-21T23:50:38.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79400K .......... .......... .......... .......... .......... 47% 51.5M 3s 2024-09-21T23:50:38.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79450K .......... .......... .......... .......... .......... 47% 32.0M 3s 2024-09-21T23:50:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79500K .......... .......... .......... .......... .......... 47% 25.9M 3s 2024-09-21T23:50:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79550K .......... .......... .......... .......... .......... 47% 23.7M 3s 2024-09-21T23:50:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79600K .......... .......... .......... .......... .......... 47% 24.0M 3s 2024-09-21T23:50:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79650K .......... .......... .......... .......... .......... 47% 25.5M 3s 2024-09-21T23:50:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79700K .......... .......... .......... .......... .......... 47% 48.1M 3s 2024-09-21T23:50:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79750K .......... .......... .......... .......... .......... 47% 28.0M 3s 2024-09-21T23:50:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79800K .......... .......... .......... .......... .......... 47% 31.9M 3s 2024-09-21T23:50:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79850K .......... .......... .......... .......... .......... 47% 22.6M 3s 2024-09-21T23:50:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79900K .......... .......... .......... .......... .......... 47% 55.7M 3s 2024-09-21T23:50:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 79950K .......... .......... .......... .......... .......... 47% 23.6M 3s 2024-09-21T23:50:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80000K .......... .......... .......... .......... .......... 47% 25.9M 3s 2024-09-21T23:50:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80050K .......... .......... .......... .......... .......... 47% 35.8M 3s 2024-09-21T23:50:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80100K .......... .......... .......... .......... .......... 47% 24.1M 3s 2024-09-21T23:50:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80150K .......... .......... .......... .......... .......... 47% 24.5M 3s 2024-09-21T23:50:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80200K .......... .......... .......... .......... .......... 47% 44.0M 3s 2024-09-21T23:50:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80250K .......... .......... .......... .......... .......... 47% 25.5M 3s 2024-09-21T23:50:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80300K .......... .......... .......... .......... .......... 47% 34.7M 3s 2024-09-21T23:50:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80350K .......... .......... .......... .......... .......... 47% 23.9M 3s 2024-09-21T23:50:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80400K .......... .......... .......... .......... .......... 47% 24.9M 3s 2024-09-21T23:50:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80450K .......... .......... .......... .......... .......... 47% 32.8M 3s 2024-09-21T23:50:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80500K .......... .......... .......... .......... .......... 47% 25.9M 3s 2024-09-21T23:50:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80550K .......... .......... .......... .......... .......... 47% 34.2M 3s 2024-09-21T23:50:38.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80600K .......... .......... .......... .......... .......... 47% 26.5M 3s 2024-09-21T23:50:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80650K .......... .......... .......... .......... .......... 47% 22.4M 3s 2024-09-21T23:50:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80700K .......... .......... .......... .......... .......... 47% 57.7M 3s 2024-09-21T23:50:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80750K .......... .......... .......... .......... .......... 47% 23.6M 3s 2024-09-21T23:50:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80800K .......... .......... .......... .......... .......... 47% 23.2M 3s 2024-09-21T23:50:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80850K .......... .......... .......... .......... .......... 47% 35.8M 3s 2024-09-21T23:50:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80900K .......... .......... .......... .......... .......... 48% 25.2M 3s 2024-09-21T23:50:38.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 80950K .......... .......... .......... .......... .......... 48% 44.9M 3s 2024-09-21T23:50:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81000K .......... .......... .......... .......... .......... 48% 25.0M 3s 2024-09-21T23:50:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81050K .......... .......... .......... .......... .......... 48% 37.2M 3s 2024-09-21T23:50:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81100K .......... .......... .......... .......... .......... 48% 25.3M 3s 2024-09-21T23:50:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81150K .......... .......... .......... .......... .......... 48% 23.4M 3s 2024-09-21T23:50:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81200K .......... .......... .......... .......... .......... 48% 24.0M 3s 2024-09-21T23:50:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81250K .......... .......... .......... .......... .......... 48% 51.6M 3s 2024-09-21T23:50:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81300K .......... .......... .......... .......... .......... 48% 33.6M 3s 2024-09-21T23:50:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81350K .......... .......... .......... .......... .......... 48% 23.9M 3s 2024-09-21T23:50:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81400K .......... .......... .......... .......... .......... 48% 24.9M 3s 2024-09-21T23:50:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81450K .......... .......... .......... .......... .......... 48% 49.2M 3s 2024-09-21T23:50:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81500K .......... .......... .......... .......... .......... 48% 24.4M 3s 2024-09-21T23:50:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81550K .......... .......... .......... .......... .......... 48% 25.3M 3s 2024-09-21T23:50:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81600K .......... .......... .......... .......... .......... 48% 34.5M 3s 2024-09-21T23:50:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81650K .......... .......... .......... .......... .......... 48% 25.5M 3s 2024-09-21T23:50:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81700K .......... .......... .......... .......... .......... 48% 51.2M 3s 2024-09-21T23:50:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81750K .......... .......... .......... .......... .......... 48% 23.4M 3s 2024-09-21T23:50:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81800K .......... .......... .......... .......... .......... 48% 34.2M 3s 2024-09-21T23:50:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81850K .......... .......... .......... .......... .......... 48% 25.4M 3s 2024-09-21T23:50:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81900K .......... .......... .......... .......... .......... 48% 24.6M 3s 2024-09-21T23:50:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 81950K .......... .......... .......... .......... .......... 48% 23.1M 3s 2024-09-21T23:50:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82000K .......... .......... .......... .......... .......... 48% 45.5M 3s 2024-09-21T23:50:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82050K .......... .......... .......... .......... .......... 48% 23.8M 3s 2024-09-21T23:50:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82100K .......... .......... .......... .......... .......... 48% 36.7M 3s 2024-09-21T23:50:38.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82150K .......... .......... .......... .......... .......... 48% 24.1M 3s 2024-09-21T23:50:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82200K .......... .......... .......... .......... .......... 48% 51.4M 3s 2024-09-21T23:50:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82250K .......... .......... .......... .......... .......... 48% 23.0M 3s 2024-09-21T23:50:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82300K .......... .......... .......... .......... .......... 48% 25.4M 3s 2024-09-21T23:50:38.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82350K .......... .......... .......... .......... .......... 48% 23.3M 3s 2024-09-21T23:50:38.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82400K .......... .......... .......... .......... .......... 48% 29.0M 3s 2024-09-21T23:50:38.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82450K .......... .......... .......... .......... .......... 48% 31.4M 3s 2024-09-21T23:50:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82500K .......... .......... .......... .......... .......... 48% 48.3M 3s 2024-09-21T23:50:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82550K .......... .......... .......... .......... .......... 49% 23.5M 3s 2024-09-21T23:50:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82600K .......... .......... .......... .......... .......... 49% 34.5M 3s 2024-09-21T23:50:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82650K .......... .......... .......... .......... .......... 49% 35.3M 3s 2024-09-21T23:50:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82700K .......... .......... .......... .......... .......... 49% 33.4M 3s 2024-09-21T23:50:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82750K .......... .......... .......... .......... .......... 49% 23.1M 3s 2024-09-21T23:50:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82800K .......... .......... .......... .......... .......... 49% 25.3M 3s 2024-09-21T23:50:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82850K .......... .......... .......... .......... .......... 49% 26.7M 3s 2024-09-21T23:50:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82900K .......... .......... .......... .......... .......... 49% 31.1M 3s 2024-09-21T23:50:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 82950K .......... .......... .......... .......... .......... 49% 28.6M 3s 2024-09-21T23:50:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83000K .......... .......... .......... .......... .......... 49% 36.2M 3s 2024-09-21T23:50:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83050K .......... .......... .......... .......... .......... 49% 23.3M 3s 2024-09-21T23:50:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83100K .......... .......... .......... .......... .......... 49% 41.8M 3s 2024-09-21T23:50:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83150K .......... .......... .......... .......... .......... 49% 17.6M 3s 2024-09-21T23:50:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83200K .......... .......... .......... .......... .......... 49% 53.6M 3s 2024-09-21T23:50:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83250K .......... .......... .......... .......... .......... 49% 36.8M 3s 2024-09-21T23:50:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83300K .......... .......... .......... .......... .......... 49% 30.8M 3s 2024-09-21T23:50:38.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83350K .......... .......... .......... .......... .......... 49% 23.5M 3s 2024-09-21T23:50:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83400K .......... .......... .......... .......... .......... 49% 28.6M 3s 2024-09-21T23:50:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83450K .......... .......... .......... .......... .......... 49% 23.4M 3s 2024-09-21T23:50:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83500K .......... .......... .......... .......... .......... 49% 54.0M 3s 2024-09-21T23:50:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83550K .......... .......... .......... .......... .......... 49% 24.1M 3s 2024-09-21T23:50:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83600K .......... .......... .......... .......... .......... 49% 24.1M 3s 2024-09-21T23:50:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83650K .......... .......... .......... .......... .......... 49% 34.8M 3s 2024-09-21T23:50:38.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83700K .......... .......... .......... .......... .......... 49% 25.5M 3s 2024-09-21T23:50:38.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83750K .......... .......... .......... .......... .......... 49% 48.2M 3s 2024-09-21T23:50:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83800K .......... .......... .......... .......... .......... 49% 23.7M 3s 2024-09-21T23:50:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83850K .......... .......... .......... .......... .......... 49% 39.4M 3s 2024-09-21T23:50:38.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83900K .......... .......... .......... .......... .......... 49% 23.2M 3s 2024-09-21T23:50:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 83950K .......... .......... .......... .......... .......... 49% 24.1M 3s 2024-09-21T23:50:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84000K .......... .......... .......... .......... .......... 49% 24.0M 3s 2024-09-21T23:50:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84050K .......... .......... .......... .......... .......... 49% 55.7M 3s 2024-09-21T23:50:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84100K .......... .......... .......... .......... .......... 49% 33.3M 3s 2024-09-21T23:50:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84150K .......... .......... .......... .......... .......... 49% 28.6M 3s 2024-09-21T23:50:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84200K .......... .......... .......... .......... .......... 49% 20.3M 3s 2024-09-21T23:50:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84250K .......... .......... .......... .......... .......... 50% 50.1M 3s 2024-09-21T23:50:38.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84300K .......... .......... .......... .......... .......... 50% 24.8M 3s 2024-09-21T23:50:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84350K .......... .......... .......... .......... .......... 50% 24.5M 3s 2024-09-21T23:50:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84400K .......... .......... .......... .......... .......... 50% 37.4M 3s 2024-09-21T23:50:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84450K .......... .......... .......... .......... .......... 50% 26.0M 3s 2024-09-21T23:50:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84500K .......... .......... .......... .......... .......... 50% 41.3M 3s 2024-09-21T23:50:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84550K .......... .......... .......... .......... .......... 50% 24.3M 3s 2024-09-21T23:50:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84600K .......... .......... .......... .......... .......... 50% 31.1M 3s 2024-09-21T23:50:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84650K .......... .......... .......... .......... .......... 50% 27.5M 3s 2024-09-21T23:50:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84700K .......... .......... .......... .......... .......... 50% 24.1M 3s 2024-09-21T23:50:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84750K .......... .......... .......... .......... .......... 50% 23.9M 3s 2024-09-21T23:50:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84800K .......... .......... .......... .......... .......... 50% 52.3M 3s 2024-09-21T23:50:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84850K .......... .......... .......... .......... .......... 50% 23.8M 3s 2024-09-21T23:50:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84900K .......... .......... .......... .......... .......... 50% 34.1M 3s 2024-09-21T23:50:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 84950K .......... .......... .......... .......... .......... 50% 25.3M 3s 2024-09-21T23:50:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85000K .......... .......... .......... .......... .......... 50% 49.1M 3s 2024-09-21T23:50:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85050K .......... .......... .......... .......... .......... 50% 24.5M 3s 2024-09-21T23:50:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85100K .......... .......... .......... .......... .......... 50% 37.2M 3s 2024-09-21T23:50:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85150K .......... .......... .......... .......... .......... 50% 23.7M 3s 2024-09-21T23:50:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85200K .......... .......... .......... .......... .......... 50% 25.5M 3s 2024-09-21T23:50:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85250K .......... .......... .......... .......... .......... 50% 23.1M 3s 2024-09-21T23:50:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85300K .......... .......... .......... .......... .......... 50% 47.6M 3s 2024-09-21T23:50:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85350K .......... .......... .......... .......... .......... 50% 24.3M 3s 2024-09-21T23:50:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85400K .......... .......... .......... .......... .......... 50% 45.7M 3s 2024-09-21T23:50:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85450K .......... .......... .......... .......... .......... 50% 24.6M 3s 2024-09-21T23:50:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85500K .......... .......... .......... .......... .......... 50% 40.4M 3s 2024-09-21T23:50:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85550K .......... .......... .......... .......... .......... 50% 23.3M 3s 2024-09-21T23:50:38.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85600K .......... .......... .......... .......... .......... 50% 25.4M 3s 2024-09-21T23:50:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85650K .......... .......... .......... .......... .......... 50% 37.2M 3s 2024-09-21T23:50:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85700K .......... .......... .......... .......... .......... 50% 24.9M 3s 2024-09-21T23:50:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85750K .......... .......... .......... .......... .......... 50% 22.4M 3s 2024-09-21T23:50:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85800K .......... .......... .......... .......... .......... 50% 55.2M 3s 2024-09-21T23:50:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85850K .......... .......... .......... .......... .......... 50% 23.6M 3s 2024-09-21T23:50:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85900K .......... .......... .......... .......... .......... 50% 37.8M 3s 2024-09-21T23:50:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 85950K .......... .......... .......... .......... .......... 51% 23.1M 3s 2024-09-21T23:50:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86000K .......... .......... .......... .......... .......... 51% 25.0M 3s 2024-09-21T23:50:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86050K .......... .......... .......... .......... .......... 51% 47.9M 3s 2024-09-21T23:50:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86100K .......... .......... .......... .......... .......... 51% 25.1M 3s 2024-09-21T23:50:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86150K .......... .......... .......... .......... .......... 51% 31.4M 3s 2024-09-21T23:50:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86200K .......... .......... .......... .......... .......... 51% 26.8M 3s 2024-09-21T23:50:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86250K .......... .......... .......... .......... .......... 51% 48.8M 3s 2024-09-21T23:50:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86300K .......... .......... .......... .......... .......... 51% 23.3M 3s 2024-09-21T23:50:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86350K .......... .......... .......... .......... .......... 51% 25.8M 3s 2024-09-21T23:50:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86400K .......... .......... .......... .......... .......... 51% 33.3M 3s 2024-09-21T23:50:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86450K .......... .......... .......... .......... .......... 51% 24.5M 3s 2024-09-21T23:50:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86500K .......... .......... .......... .......... .......... 51% 25.4M 3s 2024-09-21T23:50:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86550K .......... .......... .......... .......... .......... 51% 44.5M 3s 2024-09-21T23:50:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86600K .......... .......... .......... .......... .......... 51% 25.0M 3s 2024-09-21T23:50:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86650K .......... .......... .......... .......... .......... 51% 36.9M 3s 2024-09-21T23:50:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86700K .......... .......... .......... .......... .......... 51% 24.6M 3s 2024-09-21T23:50:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86750K .......... .......... .......... .......... .......... 51% 24.5M 3s 2024-09-21T23:50:38.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86800K .......... .......... .......... .......... .......... 51% 48.6M 3s 2024-09-21T23:50:38.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86850K .......... .......... .......... .......... .......... 51% 24.4M 3s 2024-09-21T23:50:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86900K .......... .......... .......... .......... .......... 51% 33.5M 3s 2024-09-21T23:50:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 86950K .......... .......... .......... .......... .......... 51% 24.5M 3s 2024-09-21T23:50:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87000K .......... .......... .......... .......... .......... 51% 24.0M 3s 2024-09-21T23:50:38.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87050K .......... .......... .......... .......... .......... 51% 50.2M 3s 2024-09-21T23:50:38.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87100K .......... .......... .......... .......... .......... 51% 24.9M 3s 2024-09-21T23:50:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87150K .......... .......... .......... .......... .......... 51% 26.1M 3s 2024-09-21T23:50:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87200K .......... .......... .......... .......... .......... 51% 32.3M 3s 2024-09-21T23:50:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87250K .......... .......... .......... .......... .......... 51% 24.8M 3s 2024-09-21T23:50:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87300K .......... .......... .......... .......... .......... 51% 50.7M 3s 2024-09-21T23:50:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87350K .......... .......... .......... .......... .......... 51% 23.9M 3s 2024-09-21T23:50:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87400K .......... .......... .......... .......... .......... 51% 37.2M 3s 2024-09-21T23:50:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87450K .......... .......... .......... .......... .......... 51% 23.0M 3s 2024-09-21T23:50:38.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87500K .......... .......... .......... .......... .......... 51% 24.9M 3s 2024-09-21T23:50:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87550K .......... .......... .......... .......... .......... 51% 24.2M 3s 2024-09-21T23:50:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87600K .......... .......... .......... .......... .......... 51% 49.3M 3s 2024-09-21T23:50:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87650K .......... .......... .......... .......... .......... 52% 24.6M 3s 2024-09-21T23:50:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87700K .......... .......... .......... .......... .......... 52% 37.5M 3s 2024-09-21T23:50:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87750K .......... .......... .......... .......... .......... 52% 23.6M 3s 2024-09-21T23:50:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87800K .......... .......... .......... .......... .......... 52% 47.1M 3s 2024-09-21T23:50:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87850K .......... .......... .......... .......... .......... 52% 26.6M 3s 2024-09-21T23:50:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87900K .......... .......... .......... .......... .......... 52% 38.0M 3s 2024-09-21T23:50:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 87950K .......... .......... .......... .......... .......... 52% 23.8M 3s 2024-09-21T23:50:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88000K .......... .......... .......... .......... .......... 52% 22.6M 3s 2024-09-21T23:50:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88050K .......... .......... .......... .......... .......... 52% 25.3M 3s 2024-09-21T23:50:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88100K .......... .......... .......... .......... .......... 52% 53.7M 3s 2024-09-21T23:50:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88150K .......... .......... .......... .......... .......... 52% 22.4M 3s 2024-09-21T23:50:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88200K .......... .......... .......... .......... .......... 52% 38.4M 3s 2024-09-21T23:50:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88250K .......... .......... .......... .......... .......... 52% 23.3M 3s 2024-09-21T23:50:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88300K .......... .......... .......... .......... .......... 52% 50.0M 3s 2024-09-21T23:50:38.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88350K .......... .......... .......... .......... .......... 52% 24.6M 3s 2024-09-21T23:50:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88400K .......... .......... .......... .......... .......... 52% 24.4M 3s 2024-09-21T23:50:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88450K .......... .......... .......... .......... .......... 52% 36.3M 3s 2024-09-21T23:50:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88500K .......... .......... .......... .......... .......... 52% 25.1M 3s 2024-09-21T23:50:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88550K .......... .......... .......... .......... .......... 52% 22.9M 3s 2024-09-21T23:50:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88600K .......... .......... .......... .......... .......... 52% 53.8M 3s 2024-09-21T23:50:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88650K .......... .......... .......... .......... .......... 52% 36.2M 3s 2024-09-21T23:50:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88700K .......... .......... .......... .......... .......... 52% 24.1M 3s 2024-09-21T23:50:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88750K .......... .......... .......... .......... .......... 52% 23.7M 3s 2024-09-21T23:50:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88800K .......... .......... .......... .......... .......... 52% 24.4M 3s 2024-09-21T23:50:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88850K .......... .......... .......... .......... .......... 52% 51.2M 3s 2024-09-21T23:50:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88900K .......... .......... .......... .......... .......... 52% 25.2M 3s 2024-09-21T23:50:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 88950K .......... .......... .......... .......... .......... 52% 36.8M 3s 2024-09-21T23:50:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89000K .......... .......... .......... .......... .......... 52% 23.5M 3s 2024-09-21T23:50:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89050K .......... .......... .......... .......... .......... 52% 48.1M 3s 2024-09-21T23:50:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89100K .......... .......... .......... .......... .......... 52% 25.1M 3s 2024-09-21T23:50:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89150K .......... .......... .......... .......... .......... 52% 29.2M 3s 2024-09-21T23:50:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89200K .......... .......... .......... .......... .......... 52% 26.6M 3s 2024-09-21T23:50:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89250K .......... .......... .......... .......... .......... 52% 23.9M 3s 2024-09-21T23:50:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89300K .......... .......... .......... .......... .......... 53% 27.2M 3s 2024-09-21T23:50:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89350K .......... .......... .......... .......... .......... 53% 44.3M 3s 2024-09-21T23:50:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89400K .......... .......... .......... .......... .......... 53% 23.9M 3s 2024-09-21T23:50:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89450K .......... .......... .......... .......... .......... 53% 37.5M 3s 2024-09-21T23:50:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89500K .......... .......... .......... .......... .......... 53% 24.7M 3s 2024-09-21T23:50:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89550K .......... .......... .......... .......... .......... 53% 23.8M 3s 2024-09-21T23:50:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89600K .......... .......... .......... .......... .......... 53% 48.7M 3s 2024-09-21T23:50:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89650K .......... .......... .......... .......... .......... 53% 25.6M 3s 2024-09-21T23:50:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89700K .......... .......... .......... .......... .......... 53% 33.1M 3s 2024-09-21T23:50:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89750K .......... .......... .......... .......... .......... 53% 24.1M 3s 2024-09-21T23:50:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89800K .......... .......... .......... .......... .......... 53% 24.8M 3s 2024-09-21T23:50:38.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89850K .......... .......... .......... .......... .......... 53% 47.9M 3s 2024-09-21T23:50:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89900K .......... .......... .......... .......... .......... 53% 23.8M 3s 2024-09-21T23:50:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 89950K .......... .......... .......... .......... .......... 53% 25.4M 3s 2024-09-21T23:50:38.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90000K .......... .......... .......... .......... .......... 53% 35.7M 3s 2024-09-21T23:50:38.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90050K .......... .......... .......... .......... .......... 53% 25.0M 3s 2024-09-21T23:50:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90100K .......... .......... .......... .......... .......... 53% 47.3M 3s 2024-09-21T23:50:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90150K .......... .......... .......... .......... .......... 53% 24.5M 3s 2024-09-21T23:50:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90200K .......... .......... .......... .......... .......... 53% 33.7M 3s 2024-09-21T23:50:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90250K .......... .......... .......... .......... .......... 53% 24.3M 3s 2024-09-21T23:50:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90300K .......... .......... .......... .......... .......... 53% 24.9M 3s 2024-09-21T23:50:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90350K .......... .......... .......... .......... .......... 53% 23.6M 3s 2024-09-21T23:50:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90400K .......... .......... .......... .......... .......... 53% 56.3M 3s 2024-09-21T23:50:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90450K .......... .......... .......... .......... .......... 53% 24.2M 3s 2024-09-21T23:50:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90500K .......... .......... .......... .......... .......... 53% 34.4M 3s 2024-09-21T23:50:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90550K .......... .......... .......... .......... .......... 53% 30.9M 3s 2024-09-21T23:50:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90600K .......... .......... .......... .......... .......... 53% 33.7M 3s 2024-09-21T23:50:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90650K .......... .......... .......... .......... .......... 53% 24.8M 3s 2024-09-21T23:50:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90700K .......... .......... .......... .......... .......... 53% 46.1M 3s 2024-09-21T23:50:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90750K .......... .......... .......... .......... .......... 53% 15.9M 3s 2024-09-21T23:50:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90800K .......... .......... .......... .......... .......... 53% 51.6M 3s 2024-09-21T23:50:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90850K .......... .......... .......... .......... .......... 53% 25.3M 3s 2024-09-21T23:50:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90900K .......... .......... .......... .......... .......... 53% 35.6M 3s 2024-09-21T23:50:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 90950K .......... .......... .......... .......... .......... 53% 24.2M 3s 2024-09-21T23:50:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91000K .......... .......... .......... .......... .......... 54% 27.0M 3s 2024-09-21T23:50:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91050K .......... .......... .......... .......... .......... 54% 34.1M 3s 2024-09-21T23:50:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91100K .......... .......... .......... .......... .......... 54% 45.1M 3s 2024-09-21T23:50:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91150K .......... .......... .......... .......... .......... 54% 23.9M 3s 2024-09-21T23:50:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91200K .......... .......... .......... .......... .......... 54% 25.7M 3s 2024-09-21T23:50:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91250K .......... .......... .......... .......... .......... 54% 35.7M 3s 2024-09-21T23:50:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91300K .......... .......... .......... .......... .......... 54% 28.0M 3s 2024-09-21T23:50:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91350K .......... .......... .......... .......... .......... 54% 34.6M 3s 2024-09-21T23:50:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91400K .......... .......... .......... .......... .......... 54% 26.4M 3s 2024-09-21T23:50:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91450K .......... .......... .......... .......... .......... 54% 33.7M 3s 2024-09-21T23:50:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91500K .......... .......... .......... .......... .......... 54% 26.0M 3s 2024-09-21T23:50:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91550K .......... .......... .......... .......... .......... 54% 26.9M 3s 2024-09-21T23:50:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91600K .......... .......... .......... .......... .......... 54% 22.2M 3s 2024-09-21T23:50:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91650K .......... .......... .......... .......... .......... 54% 46.2M 3s 2024-09-21T23:50:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91700K .......... .......... .......... .......... .......... 54% 24.6M 3s 2024-09-21T23:50:38.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91750K .......... .......... .......... .......... .......... 54% 34.8M 3s 2024-09-21T23:50:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91800K .......... .......... .......... .......... .......... 54% 26.8M 3s 2024-09-21T23:50:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91850K .......... .......... .......... .......... .......... 54% 42.6M 3s 2024-09-21T23:50:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91900K .......... .......... .......... .......... .......... 54% 24.4M 3s 2024-09-21T23:50:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 91950K .......... .......... .......... .......... .......... 54% 24.7M 3s 2024-09-21T23:50:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92000K .......... .......... .......... .......... .......... 54% 32.5M 3s 2024-09-21T23:50:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92050K .......... .......... .......... .......... .......... 54% 22.5M 3s 2024-09-21T23:50:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92100K .......... .......... .......... .......... .......... 54% 27.9M 3s 2024-09-21T23:50:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92150K .......... .......... .......... .......... .......... 54% 45.2M 3s 2024-09-21T23:50:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92200K .......... .......... .......... .......... .......... 54% 23.8M 3s 2024-09-21T23:50:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92250K .......... .......... .......... .......... .......... 54% 41.1M 3s 2024-09-21T23:50:38.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92300K .......... .......... .......... .......... .......... 54% 24.0M 3s 2024-09-21T23:50:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92350K .......... .......... .......... .......... .......... 54% 24.1M 3s 2024-09-21T23:50:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92400K .......... .......... .......... .......... .......... 54% 46.8M 3s 2024-09-21T23:50:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92450K .......... .......... .......... .......... .......... 54% 24.7M 3s 2024-09-21T23:50:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92500K .......... .......... .......... .......... .......... 54% 38.0M 3s 2024-09-21T23:50:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92550K .......... .......... .......... .......... .......... 54% 22.9M 3s 2024-09-21T23:50:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92600K .......... .......... .......... .......... .......... 54% 23.8M 3s 2024-09-21T23:50:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92650K .......... .......... .......... .......... .......... 54% 62.3M 3s 2024-09-21T23:50:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92700K .......... .......... .......... .......... .......... 55% 31.8M 3s 2024-09-21T23:50:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92750K .......... .......... .......... .......... .......... 55% 19.7M 3s 2024-09-21T23:50:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92800K .......... .......... .......... .......... .......... 55% 33.3M 3s 2024-09-21T23:50:38.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92850K .......... .......... .......... .......... .......... 55% 24.8M 3s 2024-09-21T23:50:38.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92900K .......... .......... .......... .......... .......... 55% 46.8M 3s 2024-09-21T23:50:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 92950K .......... .......... .......... .......... .......... 55% 24.3M 3s 2024-09-21T23:50:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93000K .......... .......... .......... .......... .......... 55% 36.6M 3s 2024-09-21T23:50:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93050K .......... .......... .......... .......... .......... 55% 24.1M 3s 2024-09-21T23:50:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93100K .......... .......... .......... .......... .......... 55% 23.3M 3s 2024-09-21T23:50:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93150K .......... .......... .......... .......... .......... 55% 24.6M 3s 2024-09-21T23:50:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93200K .......... .......... .......... .......... .......... 55% 51.7M 3s 2024-09-21T23:50:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93250K .......... .......... .......... .......... .......... 55% 24.1M 3s 2024-09-21T23:50:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93300K .......... .......... .......... .......... .......... 55% 34.8M 3s 2024-09-21T23:50:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93350K .......... .......... .......... .......... .......... 55% 25.2M 3s 2024-09-21T23:50:38.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93400K .......... .......... .......... .......... .......... 55% 49.0M 3s 2024-09-21T23:50:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93450K .......... .......... .......... .......... .......... 55% 23.6M 3s 2024-09-21T23:50:38.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93500K .......... .......... .......... .......... .......... 55% 27.4M 3s 2024-09-21T23:50:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93550K .......... .......... .......... .......... .......... 55% 30.2M 3s 2024-09-21T23:50:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93600K .......... .......... .......... .......... .......... 55% 24.8M 3s 2024-09-21T23:50:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93650K .......... .......... .......... .......... .......... 55% 23.3M 3s 2024-09-21T23:50:38.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93700K .......... .......... .......... .......... .......... 55% 50.9M 3s 2024-09-21T23:50:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93750K .......... .......... .......... .......... .......... 55% 34.3M 3s 2024-09-21T23:50:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93800K .......... .......... .......... .......... .......... 55% 24.5M 3s 2024-09-21T23:50:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93850K .......... .......... .......... .......... .......... 55% 25.6M 3s 2024-09-21T23:50:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93900K .......... .......... .......... .......... .......... 55% 47.9M 3s 2024-09-21T23:50:38.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 93950K .......... .......... .......... .......... .......... 55% 23.7M 3s 2024-09-21T23:50:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94000K .......... .......... .......... .......... .......... 55% 24.7M 3s 2024-09-21T23:50:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94050K .......... .......... .......... .......... .......... 55% 37.4M 3s 2024-09-21T23:50:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94100K .......... .......... .......... .......... .......... 55% 24.9M 3s 2024-09-21T23:50:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94150K .......... .......... .......... .......... .......... 55% 44.5M 3s 2024-09-21T23:50:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94200K .......... .......... .......... .......... .......... 55% 25.3M 3s 2024-09-21T23:50:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94250K .......... .......... .......... .......... .......... 55% 32.1M 3s 2024-09-21T23:50:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94300K .......... .......... .......... .......... .......... 55% 24.6M 3s 2024-09-21T23:50:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94350K .......... .......... .......... .......... .......... 56% 24.9M 3s 2024-09-21T23:50:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94400K .......... .......... .......... .......... .......... 56% 24.6M 3s 2024-09-21T23:50:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94450K .......... .......... .......... .......... .......... 56% 48.9M 3s 2024-09-21T23:50:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94500K .......... .......... .......... .......... .......... 56% 25.4M 3s 2024-09-21T23:50:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94550K .......... .......... .......... .......... .......... 56% 34.0M 3s 2024-09-21T23:50:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94600K .......... .......... .......... .......... .......... 56% 24.9M 3s 2024-09-21T23:50:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94650K .......... .......... .......... .......... .......... 56% 47.1M 3s 2024-09-21T23:50:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94700K .......... .......... .......... .......... .......... 56% 24.4M 3s 2024-09-21T23:50:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94750K .......... .......... .......... .......... .......... 56% 24.4M 3s 2024-09-21T23:50:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94800K .......... .......... .......... .......... .......... 56% 32.6M 3s 2024-09-21T23:50:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94850K .......... .......... .......... .......... .......... 56% 25.3M 3s 2024-09-21T23:50:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94900K .......... .......... .......... .......... .......... 56% 27.1M 3s 2024-09-21T23:50:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 94950K .......... .......... .......... .......... .......... 56% 40.8M 3s 2024-09-21T23:50:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95000K .......... .......... .......... .......... .......... 56% 23.9M 3s 2024-09-21T23:50:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95050K .......... .......... .......... .......... .......... 56% 36.7M 3s 2024-09-21T23:50:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95100K .......... .......... .......... .......... .......... 56% 25.7M 3s 2024-09-21T23:50:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95150K .......... .......... .......... .......... .......... 56% 24.5M 3s 2024-09-21T23:50:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95200K .......... .......... .......... .......... .......... 56% 44.8M 3s 2024-09-21T23:50:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95250K .......... .......... .......... .......... .......... 56% 25.1M 3s 2024-09-21T23:50:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95300K .......... .......... .......... .......... .......... 56% 39.2M 3s 2024-09-21T23:50:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95350K .......... .......... .......... .......... .......... 56% 21.9M 3s 2024-09-21T23:50:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95400K .......... .......... .......... .......... .......... 56% 26.0M 3s 2024-09-21T23:50:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95450K .......... .......... .......... .......... .......... 56% 48.2M 3s 2024-09-21T23:50:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95500K .......... .......... .......... .......... .......... 56% 24.6M 3s 2024-09-21T23:50:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95550K .......... .......... .......... .......... .......... 56% 27.2M 3s 2024-09-21T23:50:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95600K .......... .......... .......... .......... .......... 56% 27.9M 3s 2024-09-21T23:50:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95650K .......... .......... .......... .......... .......... 56% 26.6M 3s 2024-09-21T23:50:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95700K .......... .......... .......... .......... .......... 56% 47.6M 3s 2024-09-21T23:50:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95750K .......... .......... .......... .......... .......... 56% 31.0M 3s 2024-09-21T23:50:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95800K .......... .......... .......... .......... .......... 56% 28.0M 3s 2024-09-21T23:50:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95850K .......... .......... .......... .......... .......... 56% 24.4M 3s 2024-09-21T23:50:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95900K .......... .......... .......... .......... .......... 56% 23.8M 3s 2024-09-21T23:50:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 95950K .......... .......... .......... .......... .......... 56% 24.5M 3s 2024-09-21T23:50:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96000K .......... .......... .......... .......... .......... 56% 53.2M 3s 2024-09-21T23:50:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96050K .......... .......... .......... .......... .......... 57% 22.5M 3s 2024-09-21T23:50:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96100K .......... .......... .......... .......... .......... 57% 39.5M 3s 2024-09-21T23:50:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96150K .......... .......... .......... .......... .......... 57% 23.8M 3s 2024-09-21T23:50:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96200K .......... .......... .......... .......... .......... 57% 45.0M 3s 2024-09-21T23:50:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96250K .......... .......... .......... .......... .......... 57% 24.3M 3s 2024-09-21T23:50:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96300K .......... .......... .......... .......... .......... 57% 40.3M 3s 2024-09-21T23:50:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96350K .......... .......... .......... .......... .......... 57% 23.4M 3s 2024-09-21T23:50:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96400K .......... .......... .......... .......... .......... 57% 26.2M 3s 2024-09-21T23:50:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96450K .......... .......... .......... .......... .......... 57% 40.8M 3s 2024-09-21T23:50:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96500K .......... .......... .......... .......... .......... 57% 36.3M 3s 2024-09-21T23:50:38.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96550K .......... .......... .......... .......... .......... 57% 25.6M 3s 2024-09-21T23:50:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96600K .......... .......... .......... .......... .......... 57% 23.8M 3s 2024-09-21T23:50:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96650K .......... .......... .......... .......... .......... 57% 24.7M 3s 2024-09-21T23:50:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96700K .......... .......... .......... .......... .......... 57% 49.3M 3s 2024-09-21T23:50:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96750K .......... .......... .......... .......... .......... 57% 23.2M 2s 2024-09-21T23:50:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96800K .......... .......... .......... .......... .......... 57% 34.1M 2s 2024-09-21T23:50:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96850K .......... .......... .......... .......... .......... 57% 26.4M 2s 2024-09-21T23:50:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96900K .......... .......... .......... .......... .......... 57% 25.1M 2s 2024-09-21T23:50:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 96950K .......... .......... .......... .......... .......... 57% 45.2M 2s 2024-09-21T23:50:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97000K .......... .......... .......... .......... .......... 57% 24.2M 2s 2024-09-21T23:50:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97050K .......... .......... .......... .......... .......... 57% 35.0M 2s 2024-09-21T23:50:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97100K .......... .......... .......... .......... .......... 57% 25.0M 2s 2024-09-21T23:50:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97150K .......... .......... .......... .......... .......... 57% 23.5M 2s 2024-09-21T23:50:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97200K .......... .......... .......... .......... .......... 57% 23.8M 2s 2024-09-21T23:50:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97250K .......... .......... .......... .......... .......... 57% 50.5M 2s 2024-09-21T23:50:38.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97300K .......... .......... .......... .......... .......... 57% 23.6M 2s 2024-09-21T23:50:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97350K .......... .......... .......... .......... .......... 57% 37.3M 2s 2024-09-21T23:50:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97400K .......... .......... .......... .......... .......... 57% 25.5M 2s 2024-09-21T23:50:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97450K .......... .......... .......... .......... .......... 57% 44.3M 2s 2024-09-21T23:50:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97500K .......... .......... .......... .......... .......... 57% 25.0M 2s 2024-09-21T23:50:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97550K .......... .......... .......... .......... .......... 57% 24.3M 2s 2024-09-21T23:50:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97600K .......... .......... .......... .......... .......... 57% 37.0M 2s 2024-09-21T23:50:38.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97650K .......... .......... .......... .......... .......... 57% 22.8M 2s 2024-09-21T23:50:38.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97700K .......... .......... .......... .......... .......... 57% 25.5M 2s 2024-09-21T23:50:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97750K .......... .......... .......... .......... .......... 58% 50.4M 2s 2024-09-21T23:50:38.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97800K .......... .......... .......... .......... .......... 58% 31.2M 2s 2024-09-21T23:50:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97850K .......... .......... .......... .......... .......... 58% 34.3M 2s 2024-09-21T23:50:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97900K .......... .......... .......... .......... .......... 58% 20.0M 2s 2024-09-21T23:50:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 97950K .......... .......... .......... .......... .......... 58% 25.3M 2s 2024-09-21T23:50:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98000K .......... .......... .......... .......... .......... 58% 44.8M 2s 2024-09-21T23:50:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98050K .......... .......... .......... .......... .......... 58% 25.2M 2s 2024-09-21T23:50:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98100K .......... .......... .......... .......... .......... 58% 36.3M 2s 2024-09-21T23:50:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98150K .......... .......... .......... .......... .......... 58% 24.2M 2s 2024-09-21T23:50:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98200K .......... .......... .......... .......... .......... 58% 23.3M 2s 2024-09-21T23:50:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98250K .......... .......... .......... .......... .......... 58% 49.4M 2s 2024-09-21T23:50:38.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98300K .......... .......... .......... .......... .......... 58% 12.4M 2s 2024-09-21T23:50:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98350K .......... .......... .......... .......... .......... 58% 51.4M 2s 2024-09-21T23:50:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98400K .......... .......... .......... .......... .......... 58% 64.5M 2s 2024-09-21T23:50:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98450K .......... .......... .......... .......... .......... 58% 26.4M 2s 2024-09-21T23:50:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98500K .......... .......... .......... .......... .......... 58% 60.2M 2s 2024-09-21T23:50:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98550K .......... .......... .......... .......... .......... 58% 24.3M 2s 2024-09-21T23:50:38.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98600K .......... .......... .......... .......... .......... 58% 36.8M 2s 2024-09-21T23:50:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98650K .......... .......... .......... .......... .......... 58% 25.1M 2s 2024-09-21T23:50:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98700K .......... .......... .......... .......... .......... 58% 46.7M 2s 2024-09-21T23:50:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98750K .......... .......... .......... .......... .......... 58% 15.5M 2s 2024-09-21T23:50:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98800K .......... .......... .......... .......... .......... 58% 55.1M 2s 2024-09-21T23:50:38.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98850K .......... .......... .......... .......... .......... 58% 32.8M 2s 2024-09-21T23:50:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98900K .......... .......... .......... .......... .......... 58% 25.3M 2s 2024-09-21T23:50:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 98950K .......... .......... .......... .......... .......... 58% 23.8M 2s 2024-09-21T23:50:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99000K .......... .......... .......... .......... .......... 58% 50.0M 2s 2024-09-21T23:50:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99050K .......... .......... .......... .......... .......... 58% 25.0M 2s 2024-09-21T23:50:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99100K .......... .......... .......... .......... .......... 58% 36.8M 2s 2024-09-21T23:50:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99150K .......... .......... .......... .......... .......... 58% 23.6M 2s 2024-09-21T23:50:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99200K .......... .......... .......... .......... .......... 58% 25.9M 2s 2024-09-21T23:50:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99250K .......... .......... .......... .......... .......... 58% 46.9M 2s 2024-09-21T23:50:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99300K .......... .......... .......... .......... .......... 58% 24.4M 2s 2024-09-21T23:50:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99350K .......... .......... .......... .......... .......... 58% 33.1M 2s 2024-09-21T23:50:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99400K .......... .......... .......... .......... .......... 58% 25.6M 2s 2024-09-21T23:50:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99450K .......... .......... .......... .......... .......... 59% 23.7M 2s 2024-09-21T23:50:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99500K .......... .......... .......... .......... .......... 59% 48.3M 2s 2024-09-21T23:50:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99550K .......... .......... .......... .......... .......... 59% 23.9M 2s 2024-09-21T23:50:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99600K .......... .......... .......... .......... .......... 59% 24.7M 2s 2024-09-21T23:50:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99650K .......... .......... .......... .......... .......... 59% 36.8M 2s 2024-09-21T23:50:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99700K .......... .......... .......... .......... .......... 59% 35.7M 2s 2024-09-21T23:50:38.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99750K .......... .......... .......... .......... .......... 59% 33.3M 2s 2024-09-21T23:50:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99800K .......... .......... .......... .......... .......... 59% 22.6M 2s 2024-09-21T23:50:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99850K .......... .......... .......... .......... .......... 59% 38.1M 2s 2024-09-21T23:50:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99900K .......... .......... .......... .......... .......... 59% 26.8M 2s 2024-09-21T23:50:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 99950K .......... .......... .......... .......... .......... 59% 20.8M 2s 2024-09-21T23:50:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100000K .......... .......... .......... .......... .......... 59% 26.4M 2s 2024-09-21T23:50:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100050K .......... .......... .......... .......... .......... 59% 43.2M 2s 2024-09-21T23:50:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100100K .......... .......... .......... .......... .......... 59% 25.0M 2s 2024-09-21T23:50:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100150K .......... .......... .......... .......... .......... 59% 37.4M 2s 2024-09-21T23:50:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100200K .......... .......... .......... .......... .......... 59% 29.1M 2s 2024-09-21T23:50:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100250K .......... .......... .......... .......... .......... 59% 37.9M 2s 2024-09-21T23:50:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100300K .......... .......... .......... .......... .......... 59% 24.0M 2s 2024-09-21T23:50:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100350K .......... .......... .......... .......... .......... 59% 24.0M 2s 2024-09-21T23:50:38.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100400K .......... .......... .......... .......... .......... 59% 24.4M 2s 2024-09-21T23:50:38.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100450K .......... .......... .......... .......... .......... 59% 51.2M 2s 2024-09-21T23:50:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100500K .......... .......... .......... .......... .......... 59% 19.8M 2s 2024-09-21T23:50:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100550K .......... .......... .......... .......... .......... 59% 52.2M 2s 2024-09-21T23:50:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100600K .......... .......... .......... .......... .......... 59% 23.1M 2s 2024-09-21T23:50:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100650K .......... .......... .......... .......... .......... 59% 37.2M 2s 2024-09-21T23:50:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100700K .......... .......... .......... .......... .......... 59% 49.0M 2s 2024-09-21T23:50:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100750K .......... .......... .......... .......... .......... 59% 16.2M 2s 2024-09-21T23:50:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100800K .......... .......... .......... .......... .......... 59% 47.9M 2s 2024-09-21T23:50:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100850K .......... .......... .......... .......... .......... 59% 25.5M 2s 2024-09-21T23:50:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100900K .......... .......... .......... .......... .......... 59% 33.9M 2s 2024-09-21T23:50:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:100950K .......... .......... .......... .......... .......... 59% 24.8M 2s 2024-09-21T23:50:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101000K .......... .......... .......... .......... .......... 59% 23.5M 2s 2024-09-21T23:50:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101050K .......... .......... .......... .......... .......... 59% 53.9M 2s 2024-09-21T23:50:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101100K .......... .......... .......... .......... .......... 60% 34.0M 2s 2024-09-21T23:50:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101150K .......... .......... .......... .......... .......... 60% 21.1M 2s 2024-09-21T23:50:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101200K .......... .......... .......... .......... .......... 60% 28.3M 2s 2024-09-21T23:50:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101250K .......... .......... .......... .......... .......... 60% 25.6M 2s 2024-09-21T23:50:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101300K .......... .......... .......... .......... .......... 60% 46.4M 2s 2024-09-21T23:50:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101350K .......... .......... .......... .......... .......... 60% 25.0M 2s 2024-09-21T23:50:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101400K .......... .......... .......... .......... .......... 60% 36.1M 2s 2024-09-21T23:50:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101450K .......... .......... .......... .......... .......... 60% 24.9M 2s 2024-09-21T23:50:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101500K .......... .......... .......... .......... .......... 60% 48.5M 2s 2024-09-21T23:50:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101550K .......... .......... .......... .......... .......... 60% 16.1M 2s 2024-09-21T23:50:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101600K .......... .......... .......... .......... .......... 60% 51.7M 2s 2024-09-21T23:50:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101650K .......... .......... .......... .......... .......... 60% 33.8M 2s 2024-09-21T23:50:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101700K .......... .......... .......... .......... .......... 60% 25.0M 2s 2024-09-21T23:50:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101750K .......... .......... .......... .......... .......... 60% 23.9M 2s 2024-09-21T23:50:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101800K .......... .......... .......... .......... .......... 60% 49.4M 2s 2024-09-21T23:50:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101850K .......... .......... .......... .......... .......... 60% 24.6M 2s 2024-09-21T23:50:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101900K .......... .......... .......... .......... .......... 60% 35.3M 2s 2024-09-21T23:50:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:101950K .......... .......... .......... .......... .......... 60% 24.3M 2s 2024-09-21T23:50:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102000K .......... .......... .......... .......... .......... 60% 23.8M 2s 2024-09-21T23:50:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102050K .......... .......... .......... .......... .......... 60% 48.2M 2s 2024-09-21T23:50:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102100K .......... .......... .......... .......... .......... 60% 26.3M 2s 2024-09-21T23:50:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102150K .......... .......... .......... .......... .......... 60% 33.4M 2s 2024-09-21T23:50:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102200K .......... .......... .......... .......... .......... 60% 23.4M 2s 2024-09-21T23:50:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102250K .......... .......... .......... .......... .......... 60% 24.9M 2s 2024-09-21T23:50:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102300K .......... .......... .......... .......... .......... 60% 54.7M 2s 2024-09-21T23:50:38.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102350K .......... .......... .......... .......... .......... 60% 23.2M 2s 2024-09-21T23:50:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102400K .......... .......... .......... .......... .......... 60% 25.1M 2s 2024-09-21T23:50:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102450K .......... .......... .......... .......... .......... 60% 35.5M 2s 2024-09-21T23:50:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102500K .......... .......... .......... .......... .......... 60% 23.9M 2s 2024-09-21T23:50:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102550K .......... .......... .......... .......... .......... 60% 45.5M 2s 2024-09-21T23:50:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102600K .......... .......... .......... .......... .......... 60% 25.1M 2s 2024-09-21T23:50:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102650K .......... .......... .......... .......... .......... 60% 36.6M 2s 2024-09-21T23:50:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102700K .......... .......... .......... .......... .......... 60% 25.0M 2s 2024-09-21T23:50:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102750K .......... .......... .......... .......... .......... 60% 22.3M 2s 2024-09-21T23:50:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102800K .......... .......... .......... .......... .......... 61% 24.6M 2s 2024-09-21T23:50:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102850K .......... .......... .......... .......... .......... 61% 45.2M 2s 2024-09-21T23:50:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102900K .......... .......... .......... .......... .......... 61% 30.8M 2s 2024-09-21T23:50:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:102950K .......... .......... .......... .......... .......... 61% 29.1M 2s 2024-09-21T23:50:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103000K .......... .......... .......... .......... .......... 61% 23.6M 2s 2024-09-21T23:50:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103050K .......... .......... .......... .......... .......... 61% 51.0M 2s 2024-09-21T23:50:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103100K .......... .......... .......... .......... .......... 61% 24.7M 2s 2024-09-21T23:50:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103150K .......... .......... .......... .......... .......... 61% 24.5M 2s 2024-09-21T23:50:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103200K .......... .......... .......... .......... .......... 61% 36.1M 2s 2024-09-21T23:50:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103250K .......... .......... .......... .......... .......... 61% 28.2M 2s 2024-09-21T23:50:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103300K .......... .......... .......... .......... .......... 61% 24.1M 2s 2024-09-21T23:50:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103350K .......... .......... .......... .......... .......... 61% 37.4M 2s 2024-09-21T23:50:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103400K .......... .......... .......... .......... .......... 61% 35.6M 2s 2024-09-21T23:50:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103450K .......... .......... .......... .......... .......... 61% 27.4M 2s 2024-09-21T23:50:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103500K .......... .......... .......... .......... .......... 61% 22.7M 2s 2024-09-21T23:50:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103550K .......... .......... .......... .......... .......... 61% 29.1M 2s 2024-09-21T23:50:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103600K .......... .......... .......... .......... .......... 61% 34.8M 2s 2024-09-21T23:50:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103650K .......... .......... .......... .......... .......... 61% 24.3M 2s 2024-09-21T23:50:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103700K .......... .......... .......... .......... .......... 61% 39.2M 2s 2024-09-21T23:50:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103750K .......... .......... .......... .......... .......... 61% 24.0M 2s 2024-09-21T23:50:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103800K .......... .......... .......... .......... .......... 61% 51.4M 2s 2024-09-21T23:50:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103850K .......... .......... .......... .......... .......... 61% 23.5M 2s 2024-09-21T23:50:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103900K .......... .......... .......... .......... .......... 61% 34.8M 2s 2024-09-21T23:50:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:103950K .......... .......... .......... .......... .......... 61% 24.0M 2s 2024-09-21T23:50:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104000K .......... .......... .......... .......... .......... 61% 24.8M 2s 2024-09-21T23:50:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104050K .......... .......... .......... .......... .......... 61% 26.4M 2s 2024-09-21T23:50:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104100K .......... .......... .......... .......... .......... 61% 40.0M 2s 2024-09-21T23:50:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104150K .......... .......... .......... .......... .......... 61% 24.9M 2s 2024-09-21T23:50:39.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104200K .......... .......... .......... .......... .......... 61% 37.9M 2s 2024-09-21T23:50:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104250K .......... .......... .......... .......... .......... 61% 24.1M 2s 2024-09-21T23:50:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104300K .......... .......... .......... .......... .......... 61% 48.2M 2s 2024-09-21T23:50:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104350K .......... .......... .......... .......... .......... 61% 23.9M 2s 2024-09-21T23:50:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104400K .......... .......... .......... .......... .......... 61% 25.6M 2s 2024-09-21T23:50:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104450K .......... .......... .......... .......... .......... 61% 31.6M 2s 2024-09-21T23:50:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104500K .......... .......... .......... .......... .......... 62% 25.5M 2s 2024-09-21T23:50:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104550K .......... .......... .......... .......... .......... 62% 24.6M 2s 2024-09-21T23:50:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104600K .......... .......... .......... .......... .......... 62% 46.9M 2s 2024-09-21T23:50:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104650K .......... .......... .......... .......... .......... 62% 25.6M 2s 2024-09-21T23:50:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104700K .......... .......... .......... .......... .......... 62% 36.1M 2s 2024-09-21T23:50:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104750K .......... .......... .......... .......... .......... 62% 23.7M 2s 2024-09-21T23:50:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104800K .......... .......... .......... .......... .......... 62% 25.3M 2s 2024-09-21T23:50:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104850K .......... .......... .......... .......... .......... 62% 60.0M 2s 2024-09-21T23:50:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104900K .......... .......... .......... .......... .......... 62% 30.5M 2s 2024-09-21T23:50:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:104950K .......... .......... .......... .......... .......... 62% 24.6M 2s 2024-09-21T23:50:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105000K .......... .......... .......... .......... .......... 62% 24.2M 2s 2024-09-21T23:50:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105050K .......... .......... .......... .......... .......... 62% 49.5M 2s 2024-09-21T23:50:39.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105100K .......... .......... .......... .......... .......... 62% 24.9M 2s 2024-09-21T23:50:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105150K .......... .......... .......... .......... .......... 62% 23.9M 2s 2024-09-21T23:50:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105200K .......... .......... .......... .......... .......... 62% 32.1M 2s 2024-09-21T23:50:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105250K .......... .......... .......... .......... .......... 62% 22.1M 2s 2024-09-21T23:50:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105300K .......... .......... .......... .......... .......... 62% 29.8M 2s 2024-09-21T23:50:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105350K .......... .......... .......... .......... .......... 62% 45.6M 2s 2024-09-21T23:50:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105400K .......... .......... .......... .......... .......... 62% 25.6M 2s 2024-09-21T23:50:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105450K .......... .......... .......... .......... .......... 62% 34.6M 2s 2024-09-21T23:50:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105500K .......... .......... .......... .......... .......... 62% 27.0M 2s 2024-09-21T23:50:39.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105550K .......... .......... .......... .......... .......... 62% 22.0M 2s 2024-09-21T23:50:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105600K .......... .......... .......... .......... .......... 62% 24.8M 2s 2024-09-21T23:50:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105650K .......... .......... .......... .......... .......... 62% 53.6M 2s 2024-09-21T23:50:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105700K .......... .......... .......... .......... .......... 62% 23.3M 2s 2024-09-21T23:50:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105750K .......... .......... .......... .......... .......... 62% 34.5M 2s 2024-09-21T23:50:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105800K .......... .......... .......... .......... .......... 62% 25.3M 2s 2024-09-21T23:50:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105850K .......... .......... .......... .......... .......... 62% 57.0M 2s 2024-09-21T23:50:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105900K .......... .......... .......... .......... .......... 62% 23.1M 2s 2024-09-21T23:50:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:105950K .......... .......... .......... .......... .......... 62% 24.7M 2s 2024-09-21T23:50:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106000K .......... .......... .......... .......... .......... 62% 36.3M 2s 2024-09-21T23:50:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106050K .......... .......... .......... .......... .......... 62% 23.1M 2s 2024-09-21T23:50:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106100K .......... .......... .......... .......... .......... 62% 24.1M 2s 2024-09-21T23:50:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106150K .......... .......... .......... .......... .......... 63% 48.6M 2s 2024-09-21T23:50:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106200K .......... .......... .......... .......... .......... 63% 28.3M 2s 2024-09-21T23:50:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106250K .......... .......... .......... .......... .......... 63% 28.8M 2s 2024-09-21T23:50:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106300K .......... .......... .......... .......... .......... 63% 25.7M 2s 2024-09-21T23:50:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106350K .......... .......... .......... .......... .......... 63% 22.6M 2s 2024-09-21T23:50:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106400K .......... .......... .......... .......... .......... 63% 50.9M 2s 2024-09-21T23:50:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106450K .......... .......... .......... .......... .......... 63% 37.6M 2s 2024-09-21T23:50:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106500K .......... .......... .......... .......... .......... 63% 24.0M 2s 2024-09-21T23:50:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106550K .......... .......... .......... .......... .......... 63% 23.7M 2s 2024-09-21T23:50:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106600K .......... .......... .......... .......... .......... 63% 50.5M 2s 2024-09-21T23:50:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106650K .......... .......... .......... .......... .......... 63% 24.5M 2s 2024-09-21T23:50:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106700K .......... .......... .......... .......... .......... 63% 33.3M 2s 2024-09-21T23:50:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106750K .......... .......... .......... .......... .......... 63% 23.7M 2s 2024-09-21T23:50:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106800K .......... .......... .......... .......... .......... 63% 24.8M 2s 2024-09-21T23:50:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106850K .......... .......... .......... .......... .......... 63% 25.0M 2s 2024-09-21T23:50:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106900K .......... .......... .......... .......... .......... 63% 48.1M 2s 2024-09-21T23:50:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:106950K .......... .......... .......... .......... .......... 63% 23.8M 2s 2024-09-21T23:50:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107000K .......... .......... .......... .......... .......... 63% 38.1M 2s 2024-09-21T23:50:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107050K .......... .......... .......... .......... .......... 63% 24.0M 2s 2024-09-21T23:50:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107100K .......... .......... .......... .......... .......... 63% 52.2M 2s 2024-09-21T23:50:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107150K .......... .......... .......... .......... .......... 63% 23.6M 2s 2024-09-21T23:50:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107200K .......... .......... .......... .......... .......... 63% 25.0M 2s 2024-09-21T23:50:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107250K .......... .......... .......... .......... .......... 63% 33.7M 2s 2024-09-21T23:50:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107300K .......... .......... .......... .......... .......... 63% 26.2M 2s 2024-09-21T23:50:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107350K .......... .......... .......... .......... .......... 63% 22.9M 2s 2024-09-21T23:50:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107400K .......... .......... .......... .......... .......... 63% 48.9M 2s 2024-09-21T23:50:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107450K .......... .......... .......... .......... .......... 63% 25.1M 2s 2024-09-21T23:50:39.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107500K .......... .......... .......... .......... .......... 63% 35.9M 2s 2024-09-21T23:50:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107550K .......... .......... .......... .......... .......... 63% 24.0M 2s 2024-09-21T23:50:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107600K .......... .......... .......... .......... .......... 63% 25.1M 2s 2024-09-21T23:50:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107650K .......... .......... .......... .......... .......... 63% 48.1M 2s 2024-09-21T23:50:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107700K .......... .......... .......... .......... .......... 63% 24.9M 2s 2024-09-21T23:50:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107750K .......... .......... .......... .......... .......... 63% 36.6M 2s 2024-09-21T23:50:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107800K .......... .......... .......... .......... .......... 63% 22.8M 2s 2024-09-21T23:50:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107850K .......... .......... .......... .......... .......... 64% 24.7M 2s 2024-09-21T23:50:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107900K .......... .......... .......... .......... .......... 64% 73.6M 2s 2024-09-21T23:50:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:107950K .......... .......... .......... .......... .......... 64% 21.0M 2s 2024-09-21T23:50:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108000K .......... .......... .......... .......... .......... 64% 24.3M 2s 2024-09-21T23:50:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108050K .......... .......... .......... .......... .......... 64% 36.5M 2s 2024-09-21T23:50:39.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108100K .......... .......... .......... .......... .......... 64% 26.4M 2s 2024-09-21T23:50:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108150K .......... .......... .......... .......... .......... 64% 40.1M 2s 2024-09-21T23:50:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108200K .......... .......... .......... .......... .......... 64% 25.8M 2s 2024-09-21T23:50:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108250K .......... .......... .......... .......... .......... 64% 35.8M 2s 2024-09-21T23:50:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108300K .......... .......... .......... .......... .......... 64% 24.6M 2s 2024-09-21T23:50:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108350K .......... .......... .......... .......... .......... 64% 24.2M 2s 2024-09-21T23:50:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108400K .......... .......... .......... .......... .......... 64% 23.0M 2s 2024-09-21T23:50:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108450K .......... .......... .......... .......... .......... 64% 55.7M 2s 2024-09-21T23:50:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108500K .......... .......... .......... .......... .......... 64% 23.5M 2s 2024-09-21T23:50:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108550K .......... .......... .......... .......... .......... 64% 34.1M 2s 2024-09-21T23:50:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108600K .......... .......... .......... .......... .......... 64% 25.8M 2s 2024-09-21T23:50:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108650K .......... .......... .......... .......... .......... 64% 48.8M 2s 2024-09-21T23:50:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108700K .......... .......... .......... .......... .......... 64% 24.5M 2s 2024-09-21T23:50:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108750K .......... .......... .......... .......... .......... 64% 23.6M 2s 2024-09-21T23:50:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108800K .......... .......... .......... .......... .......... 64% 39.6M 2s 2024-09-21T23:50:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108850K .......... .......... .......... .......... .......... 64% 24.1M 2s 2024-09-21T23:50:39.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108900K .......... .......... .......... .......... .......... 64% 45.1M 2s 2024-09-21T23:50:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:108950K .......... .......... .......... .......... .......... 64% 25.2M 2s 2024-09-21T23:50:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109000K .......... .......... .......... .......... .......... 64% 32.7M 2s 2024-09-21T23:50:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109050K .......... .......... .......... .......... .......... 64% 25.0M 2s 2024-09-21T23:50:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109100K .......... .......... .......... .......... .......... 64% 30.8M 2s 2024-09-21T23:50:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109150K .......... .......... .......... .......... .......... 64% 21.2M 2s 2024-09-21T23:50:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109200K .......... .......... .......... .......... .......... 64% 46.6M 2s 2024-09-21T23:50:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109250K .......... .......... .......... .......... .......... 64% 23.4M 2s 2024-09-21T23:50:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109300K .......... .......... .......... .......... .......... 64% 37.0M 2s 2024-09-21T23:50:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109350K .......... .......... .......... .......... .......... 64% 22.5M 2s 2024-09-21T23:50:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109400K .......... .......... .......... .......... .......... 64% 53.8M 2s 2024-09-21T23:50:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109450K .......... .......... .......... .......... .......... 64% 25.4M 2s 2024-09-21T23:50:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109500K .......... .......... .......... .......... .......... 64% 33.7M 2s 2024-09-21T23:50:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109550K .......... .......... .......... .......... .......... 65% 19.1M 2s 2024-09-21T23:50:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109600K .......... .......... .......... .......... .......... 65% 31.6M 2s 2024-09-21T23:50:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109650K .......... .......... .......... .......... .......... 65% 24.0M 2s 2024-09-21T23:50:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109700K .......... .......... .......... .......... .......... 65% 51.5M 2s 2024-09-21T23:50:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109750K .......... .......... .......... .......... .......... 65% 24.1M 2s 2024-09-21T23:50:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109800K .......... .......... .......... .......... .......... 65% 37.1M 2s 2024-09-21T23:50:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109850K .......... .......... .......... .......... .......... 65% 24.0M 2s 2024-09-21T23:50:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109900K .......... .......... .......... .......... .......... 65% 51.4M 2s 2024-09-21T23:50:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:109950K .......... .......... .......... .......... .......... 65% 25.3M 2s 2024-09-21T23:50:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110000K .......... .......... .......... .......... .......... 65% 23.5M 2s 2024-09-21T23:50:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110050K .......... .......... .......... .......... .......... 65% 36.3M 2s 2024-09-21T23:50:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110100K .......... .......... .......... .......... .......... 65% 22.8M 2s 2024-09-21T23:50:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110150K .......... .......... .......... .......... .......... 65% 24.5M 2s 2024-09-21T23:50:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110200K .......... .......... .......... .......... .......... 65% 56.1M 2s 2024-09-21T23:50:39.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110250K .......... .......... .......... .......... .......... 65% 22.8M 2s 2024-09-21T23:50:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110300K .......... .......... .......... .......... .......... 65% 38.4M 2s 2024-09-21T23:50:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110350K .......... .......... .......... .......... .......... 65% 24.0M 2s 2024-09-21T23:50:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110400K .......... .......... .......... .......... .......... 65% 25.9M 2s 2024-09-21T23:50:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110450K .......... .......... .......... .......... .......... 65% 46.7M 2s 2024-09-21T23:50:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110500K .......... .......... .......... .......... .......... 65% 24.2M 2s 2024-09-21T23:50:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110550K .......... .......... .......... .......... .......... 65% 33.9M 2s 2024-09-21T23:50:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110600K .......... .......... .......... .......... .......... 65% 25.2M 2s 2024-09-21T23:50:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110650K .......... .......... .......... .......... .......... 65% 23.8M 2s 2024-09-21T23:50:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110700K .......... .......... .......... .......... .......... 65% 48.6M 2s 2024-09-21T23:50:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110750K .......... .......... .......... .......... .......... 65% 24.0M 2s 2024-09-21T23:50:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110800K .......... .......... .......... .......... .......... 65% 29.6M 2s 2024-09-21T23:50:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110850K .......... .......... .......... .......... .......... 65% 27.3M 2s 2024-09-21T23:50:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110900K .......... .......... .......... .......... .......... 65% 24.8M 2s 2024-09-21T23:50:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:110950K .......... .......... .......... .......... .......... 65% 47.3M 2s 2024-09-21T23:50:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111000K .......... .......... .......... .......... .......... 65% 24.3M 2s 2024-09-21T23:50:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111050K .......... .......... .......... .......... .......... 65% 37.2M 2s 2024-09-21T23:50:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111100K .......... .......... .......... .......... .......... 65% 25.2M 2s 2024-09-21T23:50:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111150K .......... .......... .......... .......... .......... 65% 24.5M 2s 2024-09-21T23:50:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111200K .......... .......... .......... .......... .......... 65% 23.1M 2s 2024-09-21T23:50:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111250K .......... .......... .......... .......... .......... 66% 53.1M 2s 2024-09-21T23:50:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111300K .......... .......... .......... .......... .......... 66% 34.1M 2s 2024-09-21T23:50:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111350K .......... .......... .......... .......... .......... 66% 23.9M 2s 2024-09-21T23:50:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111400K .......... .......... .......... .......... .......... 66% 25.7M 2s 2024-09-21T23:50:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111450K .......... .......... .......... .......... .......... 66% 48.5M 2s 2024-09-21T23:50:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111500K .......... .......... .......... .......... .......... 66% 23.3M 2s 2024-09-21T23:50:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111550K .......... .......... .......... .......... .......... 66% 24.0M 2s 2024-09-21T23:50:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111600K .......... .......... .......... .......... .......... 66% 37.0M 2s 2024-09-21T23:50:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111650K .......... .......... .......... .......... .......... 66% 25.4M 2s 2024-09-21T23:50:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111700K .......... .......... .......... .......... .......... 66% 46.0M 2s 2024-09-21T23:50:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111750K .......... .......... .......... .......... .......... 66% 24.2M 2s 2024-09-21T23:50:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111800K .......... .......... .......... .......... .......... 66% 33.9M 2s 2024-09-21T23:50:39.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111850K .......... .......... .......... .......... .......... 66% 25.0M 2s 2024-09-21T23:50:39.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111900K .......... .......... .......... .......... .......... 66% 25.6M 2s 2024-09-21T23:50:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:111950K .......... .......... .......... .......... .......... 66% 23.5M 2s 2024-09-21T23:50:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112000K .......... .......... .......... .......... .......... 66% 43.7M 2s 2024-09-21T23:50:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112050K .......... .......... .......... .......... .......... 66% 25.7M 2s 2024-09-21T23:50:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112100K .......... .......... .......... .......... .......... 66% 39.5M 2s 2024-09-21T23:50:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112150K .......... .......... .......... .......... .......... 66% 23.0M 2s 2024-09-21T23:50:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112200K .......... .......... .......... .......... .......... 66% 49.1M 2s 2024-09-21T23:50:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112250K .......... .......... .......... .......... .......... 66% 24.8M 2s 2024-09-21T23:50:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112300K .......... .......... .......... .......... .......... 66% 36.0M 2s 2024-09-21T23:50:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112350K .......... .......... .......... .......... .......... 66% 22.5M 2s 2024-09-21T23:50:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112400K .......... .......... .......... .......... .......... 66% 25.0M 2s 2024-09-21T23:50:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112450K .......... .......... .......... .......... .......... 66% 25.9M 2s 2024-09-21T23:50:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112500K .......... .......... .......... .......... .......... 66% 47.3M 2s 2024-09-21T23:50:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112550K .......... .......... .......... .......... .......... 66% 23.8M 2s 2024-09-21T23:50:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112600K .......... .......... .......... .......... .......... 66% 34.1M 2s 2024-09-21T23:50:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112650K .......... .......... .......... .......... .......... 66% 25.5M 2s 2024-09-21T23:50:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112700K .......... .......... .......... .......... .......... 66% 48.0M 2s 2024-09-21T23:50:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112750K .......... .......... .......... .......... .......... 66% 24.2M 2s 2024-09-21T23:50:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112800K .......... .......... .......... .......... .......... 66% 21.8M 2s 2024-09-21T23:50:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112850K .......... .......... .......... .......... .......... 66% 45.7M 2s 2024-09-21T23:50:39.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112900K .......... .......... .......... .......... .......... 67% 24.4M 2s 2024-09-21T23:50:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:112950K .......... .......... .......... .......... .......... 67% 24.0M 2s 2024-09-21T23:50:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113000K .......... .......... .......... .......... .......... 67% 51.3M 2s 2024-09-21T23:50:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113050K .......... .......... .......... .......... .......... 67% 23.6M 2s 2024-09-21T23:50:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113100K .......... .......... .......... .......... .......... 67% 35.6M 2s 2024-09-21T23:50:39.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113150K .......... .......... .......... .......... .......... 67% 24.4M 2s 2024-09-21T23:50:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113200K .......... .......... .......... .......... .......... 67% 25.1M 2s 2024-09-21T23:50:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113250K .......... .......... .......... .......... .......... 67% 46.9M 2s 2024-09-21T23:50:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113300K .......... .......... .......... .......... .......... 67% 24.4M 2s 2024-09-21T23:50:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113350K .......... .......... .......... .......... .......... 67% 36.9M 2s 2024-09-21T23:50:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113400K .......... .......... .......... .......... .......... 67% 24.1M 2s 2024-09-21T23:50:39.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113450K .......... .......... .......... .......... .......... 67% 23.4M 2s 2024-09-21T23:50:39.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113500K .......... .......... .......... .......... .......... 67% 51.5M 2s 2024-09-21T23:50:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113550K .......... .......... .......... .......... .......... 67% 24.7M 2s 2024-09-21T23:50:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113600K .......... .......... .......... .......... .......... 67% 23.3M 2s 2024-09-21T23:50:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113650K .......... .......... .......... .......... .......... 67% 48.2M 2s 2024-09-21T23:50:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113700K .......... .......... .......... .......... .......... 67% 23.2M 2s 2024-09-21T23:50:39.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113750K .......... .......... .......... .......... .......... 67% 40.6M 2s 2024-09-21T23:50:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113800K .......... .......... .......... .......... .......... 67% 24.8M 2s 2024-09-21T23:50:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113850K .......... .......... .......... .......... .......... 67% 36.3M 2s 2024-09-21T23:50:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113900K .......... .......... .......... .......... .......... 67% 26.1M 2s 2024-09-21T23:50:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:113950K .......... .......... .......... .......... .......... 67% 23.5M 2s 2024-09-21T23:50:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114000K .......... .......... .......... .......... .......... 67% 45.8M 2s 2024-09-21T23:50:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114050K .......... .......... .......... .......... .......... 67% 25.8M 2s 2024-09-21T23:50:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114100K .......... .......... .......... .......... .......... 67% 32.5M 2s 2024-09-21T23:50:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114150K .......... .......... .......... .......... .......... 67% 24.5M 2s 2024-09-21T23:50:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114200K .......... .......... .......... .......... .......... 67% 24.1M 2s 2024-09-21T23:50:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114250K .......... .......... .......... .......... .......... 67% 48.3M 2s 2024-09-21T23:50:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114300K .......... .......... .......... .......... .......... 67% 25.3M 2s 2024-09-21T23:50:39.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114350K .......... .......... .......... .......... .......... 67% 24.3M 2s 2024-09-21T23:50:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114400K .......... .......... .......... .......... .......... 67% 34.4M 2s 2024-09-21T23:50:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114450K .......... .......... .......... .......... .......... 67% 25.6M 2s 2024-09-21T23:50:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114500K .......... .......... .......... .......... .......... 67% 50.4M 2s 2024-09-21T23:50:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114550K .......... .......... .......... .......... .......... 67% 24.3M 2s 2024-09-21T23:50:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114600K .......... .......... .......... .......... .......... 68% 36.7M 2s 2024-09-21T23:50:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114650K .......... .......... .......... .......... .......... 68% 23.5M 2s 2024-09-21T23:50:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114700K .......... .......... .......... .......... .......... 68% 23.1M 2s 2024-09-21T23:50:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114750K .......... .......... .......... .......... .......... 68% 29.2M 2s 2024-09-21T23:50:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114800K .......... .......... .......... .......... .......... 68% 37.0M 2s 2024-09-21T23:50:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114850K .......... .......... .......... .......... .......... 68% 25.8M 2s 2024-09-21T23:50:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114900K .......... .......... .......... .......... .......... 68% 33.1M 2s 2024-09-21T23:50:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:114950K .......... .......... .......... .......... .......... 68% 24.6M 2s 2024-09-21T23:50:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115000K .......... .......... .......... .......... .......... 68% 50.5M 2s 2024-09-21T23:50:39.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115050K .......... .......... .......... .......... .......... 68% 24.4M 2s 2024-09-21T23:50:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115100K .......... .......... .......... .......... .......... 68% 37.6M 2s 2024-09-21T23:50:39.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115150K .......... .......... .......... .......... .......... 68% 24.3M 2s 2024-09-21T23:50:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115200K .......... .......... .......... .......... .......... 68% 23.0M 2s 2024-09-21T23:50:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115250K .......... .......... .......... .......... .......... 68% 24.8M 2s 2024-09-21T23:50:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115300K .......... .......... .......... .......... .......... 68% 52.0M 2s 2024-09-21T23:50:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115350K .......... .......... .......... .......... .......... 68% 23.8M 2s 2024-09-21T23:50:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115400K .......... .......... .......... .......... .......... 68% 35.2M 2s 2024-09-21T23:50:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115450K .......... .......... .......... .......... .......... 68% 24.8M 2s 2024-09-21T23:50:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115500K .......... .......... .......... .......... .......... 68% 50.6M 2s 2024-09-21T23:50:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115550K .......... .......... .......... .......... .......... 68% 25.9M 2s 2024-09-21T23:50:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115600K .......... .......... .......... .......... .......... 68% 22.8M 2s 2024-09-21T23:50:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115650K .......... .......... .......... .......... .......... 68% 38.2M 2s 2024-09-21T23:50:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115700K .......... .......... .......... .......... .......... 68% 24.3M 2s 2024-09-21T23:50:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115750K .......... .......... .......... .......... .......... 68% 22.6M 2s 2024-09-21T23:50:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115800K .......... .......... .......... .......... .......... 68% 52.9M 2s 2024-09-21T23:50:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115850K .......... .......... .......... .......... .......... 68% 34.1M 2s 2024-09-21T23:50:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115900K .......... .......... .......... .......... .......... 68% 25.6M 2s 2024-09-21T23:50:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:115950K .......... .......... .......... .......... .......... 68% 25.1M 2s 2024-09-21T23:50:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116000K .......... .......... .......... .......... .......... 68% 23.6M 2s 2024-09-21T23:50:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116050K .......... .......... .......... .......... .......... 68% 47.2M 2s 2024-09-21T23:50:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116100K .......... .......... .......... .......... .......... 68% 25.4M 2s 2024-09-21T23:50:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116150K .......... .......... .......... .......... .......... 68% 34.1M 2s 2024-09-21T23:50:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116200K .......... .......... .......... .......... .......... 68% 24.8M 2s 2024-09-21T23:50:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116250K .......... .......... .......... .......... .......... 68% 45.8M 2s 2024-09-21T23:50:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116300K .......... .......... .......... .......... .......... 69% 25.9M 2s 2024-09-21T23:50:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116350K .......... .......... .......... .......... .......... 69% 24.3M 2s 2024-09-21T23:50:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116400K .......... .......... .......... .......... .......... 69% 30.8M 2s 2024-09-21T23:50:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116450K .......... .......... .......... .......... .......... 69% 24.7M 2s 2024-09-21T23:50:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116500K .......... .......... .......... .......... .......... 69% 26.7M 2s 2024-09-21T23:50:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116550K .......... .......... .......... .......... .......... 69% 51.0M 2s 2024-09-21T23:50:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116600K .......... .......... .......... .......... .......... 69% 22.9M 2s 2024-09-21T23:50:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116650K .......... .......... .......... .......... .......... 69% 38.7M 2s 2024-09-21T23:50:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116700K .......... .......... .......... .......... .......... 69% 24.4M 2s 2024-09-21T23:50:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116750K .......... .......... .......... .......... .......... 69% 23.8M 2s 2024-09-21T23:50:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116800K .......... .......... .......... .......... .......... 69% 49.7M 2s 2024-09-21T23:50:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116850K .......... .......... .......... .......... .......... 69% 25.0M 2s 2024-09-21T23:50:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116900K .......... .......... .......... .......... .......... 69% 31.9M 2s 2024-09-21T23:50:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:116950K .......... .......... .......... .......... .......... 69% 25.2M 2s 2024-09-21T23:50:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117000K .......... .......... .......... .......... .......... 69% 24.8M 2s 2024-09-21T23:50:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117050K .......... .......... .......... .......... .......... 69% 47.9M 2s 2024-09-21T23:50:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117100K .......... .......... .......... .......... .......... 69% 24.8M 2s 2024-09-21T23:50:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117150K .......... .......... .......... .......... .......... 69% 24.1M 2s 2024-09-21T23:50:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117200K .......... .......... .......... .......... .......... 69% 37.0M 2s 2024-09-21T23:50:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117250K .......... .......... .......... .......... .......... 69% 24.1M 2s 2024-09-21T23:50:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117300K .......... .......... .......... .......... .......... 69% 49.4M 2s 2024-09-21T23:50:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117350K .......... .......... .......... .......... .......... 69% 24.4M 2s 2024-09-21T23:50:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117400K .......... .......... .......... .......... .......... 69% 37.9M 2s 2024-09-21T23:50:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117450K .......... .......... .......... .......... .......... 69% 22.0M 2s 2024-09-21T23:50:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117500K .......... .......... .......... .......... .......... 69% 25.3M 2s 2024-09-21T23:50:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117550K .......... .......... .......... .......... .......... 69% 23.6M 2s 2024-09-21T23:50:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117600K .......... .......... .......... .......... .......... 69% 50.4M 2s 2024-09-21T23:50:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117650K .......... .......... .......... .......... .......... 69% 24.3M 2s 2024-09-21T23:50:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117700K .......... .......... .......... .......... .......... 69% 37.9M 2s 2024-09-21T23:50:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117750K .......... .......... .......... .......... .......... 69% 23.7M 2s 2024-09-21T23:50:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117800K .......... .......... .......... .......... .......... 69% 48.8M 2s 2024-09-21T23:50:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117850K .......... .......... .......... .......... .......... 69% 24.8M 2s 2024-09-21T23:50:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117900K .......... .......... .......... .......... .......... 69% 37.8M 2s 2024-09-21T23:50:39.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:117950K .......... .......... .......... .......... .......... 70% 23.1M 2s 2024-09-21T23:50:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118000K .......... .......... .......... .......... .......... 70% 23.5M 2s 2024-09-21T23:50:39.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118050K .......... .......... .......... .......... .......... 70% 24.1M 2s 2024-09-21T23:50:39.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118100K .......... .......... .......... .......... .......... 70% 57.2M 2s 2024-09-21T23:50:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118150K .......... .......... .......... .......... .......... 70% 22.2M 2s 2024-09-21T23:50:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118200K .......... .......... .......... .......... .......... 70% 36.5M 2s 2024-09-21T23:50:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118250K .......... .......... .......... .......... .......... 70% 24.9M 2s 2024-09-21T23:50:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118300K .......... .......... .......... .......... .......... 70% 50.1M 2s 2024-09-21T23:50:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118350K .......... .......... .......... .......... .......... 70% 23.4M 2s 2024-09-21T23:50:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118400K .......... .......... .......... .......... .......... 70% 25.5M 2s 2024-09-21T23:50:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118450K .......... .......... .......... .......... .......... 70% 37.2M 2s 2024-09-21T23:50:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118500K .......... .......... .......... .......... .......... 70% 25.0M 2s 2024-09-21T23:50:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118550K .......... .......... .......... .......... .......... 70% 23.0M 2s 2024-09-21T23:50:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118600K .......... .......... .......... .......... .......... 70% 52.1M 2s 2024-09-21T23:50:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118650K .......... .......... .......... .......... .......... 70% 34.8M 2s 2024-09-21T23:50:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118700K .......... .......... .......... .......... .......... 70% 25.2M 2s 2024-09-21T23:50:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118750K .......... .......... .......... .......... .......... 70% 22.7M 2s 2024-09-21T23:50:39.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118800K .......... .......... .......... .......... .......... 70% 24.8M 2s 2024-09-21T23:50:39.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118850K .......... .......... .......... .......... .......... 70% 46.8M 2s 2024-09-21T23:50:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118900K .......... .......... .......... .......... .......... 70% 28.0M 2s 2024-09-21T23:50:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:118950K .......... .......... .......... .......... .......... 70% 37.1M 2s 2024-09-21T23:50:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119000K .......... .......... .......... .......... .......... 70% 22.0M 2s 2024-09-21T23:50:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119050K .......... .......... .......... .......... .......... 70% 51.6M 2s 2024-09-21T23:50:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119100K .......... .......... .......... .......... .......... 70% 24.4M 2s 2024-09-21T23:50:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119150K .......... .......... .......... .......... .......... 70% 24.5M 2s 2024-09-21T23:50:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119200K .......... .......... .......... .......... .......... 70% 32.9M 2s 2024-09-21T23:50:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119250K .......... .......... .......... .......... .......... 70% 30.4M 2s 2024-09-21T23:50:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119300K .......... .......... .......... .......... .......... 70% 21.3M 2s 2024-09-21T23:50:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119350K .......... .......... .......... .......... .......... 70% 44.7M 2s 2024-09-21T23:50:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119400K .......... .......... .......... .......... .......... 70% 24.9M 2s 2024-09-21T23:50:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119450K .......... .......... .......... .......... .......... 70% 33.9M 2s 2024-09-21T23:50:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119500K .......... .......... .......... .......... .......... 70% 26.3M 2s 2024-09-21T23:50:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119550K .......... .......... .......... .......... .......... 70% 23.6M 2s 2024-09-21T23:50:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119600K .......... .......... .......... .......... .......... 70% 51.2M 2s 2024-09-21T23:50:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119650K .......... .......... .......... .......... .......... 71% 24.7M 2s 2024-09-21T23:50:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119700K .......... .......... .......... .......... .......... 71% 36.0M 2s 2024-09-21T23:50:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119750K .......... .......... .......... .......... .......... 71% 23.1M 2s 2024-09-21T23:50:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119800K .......... .......... .......... .......... .......... 71% 24.2M 2s 2024-09-21T23:50:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119850K .......... .......... .......... .......... .......... 71% 49.0M 2s 2024-09-21T23:50:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119900K .......... .......... .......... .......... .......... 71% 25.6M 2s 2024-09-21T23:50:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:119950K .......... .......... .......... .......... .......... 71% 23.1M 2s 2024-09-21T23:50:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120000K .......... .......... .......... .......... .......... 71% 38.8M 2s 2024-09-21T23:50:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120050K .......... .......... .......... .......... .......... 71% 24.8M 2s 2024-09-21T23:50:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120100K .......... .......... .......... .......... .......... 71% 46.3M 2s 2024-09-21T23:50:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120150K .......... .......... .......... .......... .......... 71% 24.0M 2s 2024-09-21T23:50:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120200K .......... .......... .......... .......... .......... 71% 26.8M 2s 2024-09-21T23:50:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120250K .......... .......... .......... .......... .......... 71% 32.7M 2s 2024-09-21T23:50:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120300K .......... .......... .......... .......... .......... 71% 40.2M 2s 2024-09-21T23:50:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120350K .......... .......... .......... .......... .......... 71% 20.1M 2s 2024-09-21T23:50:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120400K .......... .......... .......... .......... .......... 71% 34.1M 2s 2024-09-21T23:50:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120450K .......... .......... .......... .......... .......... 71% 27.5M 2s 2024-09-21T23:50:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120500K .......... .......... .......... .......... .......... 71% 24.5M 2s 2024-09-21T23:50:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120550K .......... .......... .......... .......... .......... 71% 39.8M 2s 2024-09-21T23:50:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120600K .......... .......... .......... .......... .......... 71% 33.3M 2s 2024-09-21T23:50:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120650K .......... .......... .......... .......... .......... 71% 22.7M 2s 2024-09-21T23:50:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120700K .......... .......... .......... .......... .......... 71% 48.2M 2s 2024-09-21T23:50:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120750K .......... .......... .......... .......... .......... 71% 22.0M 2s 2024-09-21T23:50:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120800K .......... .......... .......... .......... .......... 71% 26.3M 2s 2024-09-21T23:50:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120850K .......... .......... .......... .......... .......... 71% 38.9M 2s 2024-09-21T23:50:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120900K .......... .......... .......... .......... .......... 71% 30.6M 2s 2024-09-21T23:50:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:120950K .......... .......... .......... .......... .......... 71% 21.8M 2s 2024-09-21T23:50:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121000K .......... .......... .......... .......... .......... 71% 40.3M 2s 2024-09-21T23:50:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121050K .......... .......... .......... .......... .......... 71% 24.6M 2s 2024-09-21T23:50:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121100K .......... .......... .......... .......... .......... 71% 46.2M 2s 2024-09-21T23:50:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121150K .......... .......... .......... .......... .......... 71% 23.1M 2s 2024-09-21T23:50:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121200K .......... .......... .......... .......... .......... 71% 27.0M 2s 2024-09-21T23:50:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121250K .......... .......... .......... .......... .......... 71% 25.0M 2s 2024-09-21T23:50:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121300K .......... .......... .......... .......... .......... 71% 47.4M 2s 2024-09-21T23:50:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121350K .......... .......... .......... .......... .......... 72% 32.9M 2s 2024-09-21T23:50:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121400K .......... .......... .......... .......... .......... 72% 24.3M 2s 2024-09-21T23:50:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121450K .......... .......... .......... .......... .......... 72% 37.1M 2s 2024-09-21T23:50:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121500K .......... .......... .......... .......... .......... 72% 32.6M 2s 2024-09-21T23:50:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121550K .......... .......... .......... .......... .......... 72% 18.6M 2s 2024-09-21T23:50:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121600K .......... .......... .......... .......... .......... 72% 24.5M 2s 2024-09-21T23:50:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121650K .......... .......... .......... .......... .......... 72% 43.3M 2s 2024-09-21T23:50:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121700K .......... .......... .......... .......... .......... 72% 24.9M 2s 2024-09-21T23:50:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121750K .......... .......... .......... .......... .......... 72% 36.2M 2s 2024-09-21T23:50:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121800K .......... .......... .......... .......... .......... 72% 25.8M 2s 2024-09-21T23:50:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121850K .......... .......... .......... .......... .......... 72% 45.9M 2s 2024-09-21T23:50:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121900K .......... .......... .......... .......... .......... 72% 24.0M 2s 2024-09-21T23:50:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:121950K .......... .......... .......... .......... .......... 72% 24.3M 2s 2024-09-21T23:50:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122000K .......... .......... .......... .......... .......... 72% 23.2M 2s 2024-09-21T23:50:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122050K .......... .......... .......... .......... .......... 72% 37.6M 2s 2024-09-21T23:50:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122100K .......... .......... .......... .......... .......... 72% 24.7M 2s 2024-09-21T23:50:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122150K .......... .......... .......... .......... .......... 72% 48.3M 2s 2024-09-21T23:50:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122200K .......... .......... .......... .......... .......... 72% 23.6M 2s 2024-09-21T23:50:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122250K .......... .......... .......... .......... .......... 72% 48.0M 2s 2024-09-21T23:50:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122300K .......... .......... .......... .......... .......... 72% 22.3M 2s 2024-09-21T23:50:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122350K .......... .......... .......... .......... .......... 72% 23.6M 2s 2024-09-21T23:50:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122400K .......... .......... .......... .......... .......... 72% 48.7M 2s 2024-09-21T23:50:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122450K .......... .......... .......... .......... .......... 72% 29.5M 2s 2024-09-21T23:50:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122500K .......... .......... .......... .......... .......... 72% 20.4M 2s 2024-09-21T23:50:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122550K .......... .......... .......... .......... .......... 72% 35.0M 2s 2024-09-21T23:50:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122600K .......... .......... .......... .......... .......... 72% 25.2M 2s 2024-09-21T23:50:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122650K .......... .......... .......... .......... .......... 72% 51.0M 2s 2024-09-21T23:50:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122700K .......... .......... .......... .......... .......... 72% 24.3M 2s 2024-09-21T23:50:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122750K .......... .......... .......... .......... .......... 72% 23.1M 2s 2024-09-21T23:50:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122800K .......... .......... .......... .......... .......... 72% 37.5M 2s 2024-09-21T23:50:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122850K .......... .......... .......... .......... .......... 72% 28.8M 2s 2024-09-21T23:50:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122900K .......... .......... .......... .......... .......... 72% 37.0M 2s 2024-09-21T23:50:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:122950K .......... .......... .......... .......... .......... 72% 24.8M 2s 2024-09-21T23:50:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123000K .......... .......... .......... .......... .......... 72% 36.2M 2s 2024-09-21T23:50:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123050K .......... .......... .......... .......... .......... 73% 25.1M 2s 2024-09-21T23:50:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123100K .......... .......... .......... .......... .......... 73% 23.1M 2s 2024-09-21T23:50:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123150K .......... .......... .......... .......... .......... 73% 24.1M 2s 2024-09-21T23:50:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123200K .......... .......... .......... .......... .......... 73% 53.2M 2s 2024-09-21T23:50:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123250K .......... .......... .......... .......... .......... 73% 24.1M 2s 2024-09-21T23:50:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123300K .......... .......... .......... .......... .......... 73% 40.7M 2s 2024-09-21T23:50:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123350K .......... .......... .......... .......... .......... 73% 22.7M 2s 2024-09-21T23:50:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123400K .......... .......... .......... .......... .......... 73% 47.7M 2s 2024-09-21T23:50:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123450K .......... .......... .......... .......... .......... 73% 25.1M 2s 2024-09-21T23:50:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123500K .......... .......... .......... .......... .......... 73% 35.6M 2s 2024-09-21T23:50:39.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123550K .......... .......... .......... .......... .......... 73% 17.4M 2s 2024-09-21T23:50:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123600K .......... .......... .......... .......... .......... 73% 41.8M 2s 2024-09-21T23:50:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123650K .......... .......... .......... .......... .......... 73% 23.4M 2s 2024-09-21T23:50:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123700K .......... .......... .......... .......... .......... 73% 54.1M 2s 2024-09-21T23:50:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123750K .......... .......... .......... .......... .......... 73% 31.2M 2s 2024-09-21T23:50:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123800K .......... .......... .......... .......... .......... 73% 29.2M 2s 2024-09-21T23:50:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123850K .......... .......... .......... .......... .......... 73% 21.6M 2s 2024-09-21T23:50:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123900K .......... .......... .......... .......... .......... 73% 51.1M 2s 2024-09-21T23:50:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:123950K .......... .......... .......... .......... .......... 73% 23.7M 2s 2024-09-21T23:50:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124000K .......... .......... .......... .......... .......... 73% 25.1M 2s 2024-09-21T23:50:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124050K .......... .......... .......... .......... .......... 73% 37.2M 2s 2024-09-21T23:50:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124100K .......... .......... .......... .......... .......... 73% 24.4M 2s 2024-09-21T23:50:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124150K .......... .......... .......... .......... .......... 73% 46.8M 2s 2024-09-21T23:50:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124200K .......... .......... .......... .......... .......... 73% 24.6M 2s 2024-09-21T23:50:39.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124250K .......... .......... .......... .......... .......... 73% 33.3M 2s 2024-09-21T23:50:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124300K .......... .......... .......... .......... .......... 73% 24.9M 2s 2024-09-21T23:50:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124350K .......... .......... .......... .......... .......... 73% 24.4M 2s 2024-09-21T23:50:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124400K .......... .......... .......... .......... .......... 73% 30.2M 2s 2024-09-21T23:50:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124450K .......... .......... .......... .......... .......... 73% 40.4M 2s 2024-09-21T23:50:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124500K .......... .......... .......... .......... .......... 73% 23.1M 2s 2024-09-21T23:50:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124550K .......... .......... .......... .......... .......... 73% 34.8M 2s 2024-09-21T23:50:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124600K .......... .......... .......... .......... .......... 73% 24.5M 2s 2024-09-21T23:50:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124650K .......... .......... .......... .......... .......... 73% 59.0M 2s 2024-09-21T23:50:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124700K .......... .......... .......... .......... .......... 74% 22.8M 2s 2024-09-21T23:50:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124750K .......... .......... .......... .......... .......... 74% 24.4M 2s 2024-09-21T23:50:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124800K .......... .......... .......... .......... .......... 74% 23.3M 2s 2024-09-21T23:50:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124850K .......... .......... .......... .......... .......... 74% 36.9M 2s 2024-09-21T23:50:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124900K .......... .......... .......... .......... .......... 74% 24.6M 2s 2024-09-21T23:50:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:124950K .......... .......... .......... .......... .......... 74% 45.4M 2s 2024-09-21T23:50:39.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125000K .......... .......... .......... .......... .......... 74% 24.8M 2s 2024-09-21T23:50:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125050K .......... .......... .......... .......... .......... 74% 36.8M 2s 2024-09-21T23:50:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125100K .......... .......... .......... .......... .......... 74% 24.5M 2s 2024-09-21T23:50:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125150K .......... .......... .......... .......... .......... 74% 24.6M 1s 2024-09-21T23:50:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125200K .......... .......... .......... .......... .......... 74% 59.7M 1s 2024-09-21T23:50:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125250K .......... .......... .......... .......... .......... 74% 22.8M 1s 2024-09-21T23:50:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125300K .......... .......... .......... .......... .......... 74% 36.3M 1s 2024-09-21T23:50:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125350K .......... .......... .......... .......... .......... 74% 24.2M 1s 2024-09-21T23:50:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125400K .......... .......... .......... .......... .......... 74% 23.1M 1s 2024-09-21T23:50:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125450K .......... .......... .......... .......... .......... 74% 55.7M 1s 2024-09-21T23:50:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125500K .......... .......... .......... .......... .......... 74% 23.7M 1s 2024-09-21T23:50:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125550K .......... .......... .......... .......... .......... 74% 23.7M 1s 2024-09-21T23:50:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125600K .......... .......... .......... .......... .......... 74% 36.1M 1s 2024-09-21T23:50:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125650K .......... .......... .......... .......... .......... 74% 25.1M 1s 2024-09-21T23:50:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125700K .......... .......... .......... .......... .......... 74% 47.2M 1s 2024-09-21T23:50:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125750K .......... .......... .......... .......... .......... 74% 23.5M 1s 2024-09-21T23:50:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125800K .......... .......... .......... .......... .......... 74% 36.7M 1s 2024-09-21T23:50:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125850K .......... .......... .......... .......... .......... 74% 25.1M 1s 2024-09-21T23:50:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125900K .......... .......... .......... .......... .......... 74% 48.1M 1s 2024-09-21T23:50:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:125950K .......... .......... .......... .......... .......... 74% 15.6M 1s 2024-09-21T23:50:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126000K .......... .......... .......... .......... .......... 74% 56.7M 1s 2024-09-21T23:50:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126050K .......... .......... .......... .......... .......... 74% 29.7M 1s 2024-09-21T23:50:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126100K .......... .......... .......... .......... .......... 74% 26.5M 1s 2024-09-21T23:50:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126150K .......... .......... .......... .......... .......... 74% 24.7M 1s 2024-09-21T23:50:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126200K .......... .......... .......... .......... .......... 74% 51.7M 1s 2024-09-21T23:50:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126250K .......... .......... .......... .......... .......... 74% 23.9M 1s 2024-09-21T23:50:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126300K .......... .......... .......... .......... .......... 74% 37.1M 1s 2024-09-21T23:50:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126350K .......... .......... .......... .......... .......... 74% 24.1M 1s 2024-09-21T23:50:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126400K .......... .......... .......... .......... .......... 75% 25.2M 1s 2024-09-21T23:50:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126450K .......... .......... .......... .......... .......... 75% 43.8M 1s 2024-09-21T23:50:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126500K .......... .......... .......... .......... .......... 75% 26.1M 1s 2024-09-21T23:50:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126550K .......... .......... .......... .......... .......... 75% 31.0M 1s 2024-09-21T23:50:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126600K .......... .......... .......... .......... .......... 75% 25.3M 1s 2024-09-21T23:50:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126650K .......... .......... .......... .......... .......... 75% 25.7M 1s 2024-09-21T23:50:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126700K .......... .......... .......... .......... .......... 75% 47.1M 1s 2024-09-21T23:50:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126750K .......... .......... .......... .......... .......... 75% 23.4M 1s 2024-09-21T23:50:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126800K .......... .......... .......... .......... .......... 75% 23.9M 1s 2024-09-21T23:50:39.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126850K .......... .......... .......... .......... .......... 75% 35.6M 1s 2024-09-21T23:50:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126900K .......... .......... .......... .......... .......... 75% 25.1M 1s 2024-09-21T23:50:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:126950K .......... .......... .......... .......... .......... 75% 46.2M 1s 2024-09-21T23:50:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127000K .......... .......... .......... .......... .......... 75% 23.6M 1s 2024-09-21T23:50:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127050K .......... .......... .......... .......... .......... 75% 41.7M 1s 2024-09-21T23:50:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127100K .......... .......... .......... .......... .......... 75% 22.8M 1s 2024-09-21T23:50:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127150K .......... .......... .......... .......... .......... 75% 23.8M 1s 2024-09-21T23:50:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127200K .......... .......... .......... .......... .......... 75% 26.0M 1s 2024-09-21T23:50:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127250K .......... .......... .......... .......... .......... 75% 44.4M 1s 2024-09-21T23:50:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127300K .......... .......... .......... .......... .......... 75% 25.7M 1s 2024-09-21T23:50:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127350K .......... .......... .......... .......... .......... 75% 35.2M 1s 2024-09-21T23:50:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127400K .......... .......... .......... .......... .......... 75% 24.2M 1s 2024-09-21T23:50:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127450K .......... .......... .......... .......... .......... 75% 48.1M 1s 2024-09-21T23:50:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127500K .......... .......... .......... .......... .......... 75% 25.5M 1s 2024-09-21T23:50:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127550K .......... .......... .......... .......... .......... 75% 23.6M 1s 2024-09-21T23:50:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127600K .......... .......... .......... .......... .......... 75% 37.7M 1s 2024-09-21T23:50:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127650K .......... .......... .......... .......... .......... 75% 23.4M 1s 2024-09-21T23:50:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127700K .......... .......... .......... .......... .......... 75% 25.3M 1s 2024-09-21T23:50:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127750K .......... .......... .......... .......... .......... 75% 48.8M 1s 2024-09-21T23:50:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127800K .......... .......... .......... .......... .......... 75% 23.3M 1s 2024-09-21T23:50:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127850K .......... .......... .......... .......... .......... 75% 37.6M 1s 2024-09-21T23:50:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127900K .......... .......... .......... .......... .......... 75% 24.3M 1s 2024-09-21T23:50:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:127950K .......... .......... .......... .......... .......... 75% 24.1M 1s 2024-09-21T23:50:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128000K .......... .......... .......... .......... .......... 75% 12.2M 1s 2024-09-21T23:50:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128050K .......... .......... .......... .......... .......... 75% 126M 1s 2024-09-21T23:50:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128100K .......... .......... .......... .......... .......... 76% 142M 1s 2024-09-21T23:50:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128150K .......... .......... .......... .......... .......... 76% 30.6M 1s 2024-09-21T23:50:39.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128200K .......... .......... .......... .......... .......... 76% 22.8M 1s 2024-09-21T23:50:39.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128250K .......... .......... .......... .......... .......... 76% 57.4M 1s 2024-09-21T23:50:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128300K .......... .......... .......... .......... .......... 76% 33.4M 1s 2024-09-21T23:50:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128350K .......... .......... .......... .......... .......... 76% 18.4M 1s 2024-09-21T23:50:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128400K .......... .......... .......... .......... .......... 76% 36.8M 1s 2024-09-21T23:50:39.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128450K .......... .......... .......... .......... .......... 76% 24.7M 1s 2024-09-21T23:50:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128500K .......... .......... .......... .......... .......... 76% 47.4M 1s 2024-09-21T23:50:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128550K .......... .......... .......... .......... .......... 76% 25.4M 1s 2024-09-21T23:50:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128600K .......... .......... .......... .......... .......... 76% 35.1M 1s 2024-09-21T23:50:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128650K .......... .......... .......... .......... .......... 76% 24.5M 1s 2024-09-21T23:50:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128700K .......... .......... .......... .......... .......... 76% 47.9M 1s 2024-09-21T23:50:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128750K .......... .......... .......... .......... .......... 76% 15.7M 1s 2024-09-21T23:50:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128800K .......... .......... .......... .......... .......... 76% 57.9M 1s 2024-09-21T23:50:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128850K .......... .......... .......... .......... .......... 76% 32.5M 1s 2024-09-21T23:50:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128900K .......... .......... .......... .......... .......... 76% 24.6M 1s 2024-09-21T23:50:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:128950K .......... .......... .......... .......... .......... 76% 25.1M 1s 2024-09-21T23:50:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129000K .......... .......... .......... .......... .......... 76% 42.5M 1s 2024-09-21T23:50:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129050K .......... .......... .......... .......... .......... 76% 25.5M 1s 2024-09-21T23:50:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129100K .......... .......... .......... .......... .......... 76% 38.8M 1s 2024-09-21T23:50:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129150K .......... .......... .......... .......... .......... 76% 23.8M 1s 2024-09-21T23:50:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129200K .......... .......... .......... .......... .......... 76% 25.2M 1s 2024-09-21T23:50:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129250K .......... .......... .......... .......... .......... 76% 45.4M 1s 2024-09-21T23:50:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129300K .......... .......... .......... .......... .......... 76% 25.8M 1s 2024-09-21T23:50:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129350K .......... .......... .......... .......... .......... 76% 32.7M 1s 2024-09-21T23:50:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129400K .......... .......... .......... .......... .......... 76% 24.6M 1s 2024-09-21T23:50:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129450K .......... .......... .......... .......... .......... 76% 24.6M 1s 2024-09-21T23:50:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129500K .......... .......... .......... .......... .......... 76% 46.6M 1s 2024-09-21T23:50:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129550K .......... .......... .......... .......... .......... 76% 27.3M 1s 2024-09-21T23:50:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129600K .......... .......... .......... .......... .......... 76% 27.7M 1s 2024-09-21T23:50:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129650K .......... .......... .......... .......... .......... 76% 25.0M 1s 2024-09-21T23:50:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129700K .......... .......... .......... .......... .......... 76% 25.3M 1s 2024-09-21T23:50:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129750K .......... .......... .......... .......... .......... 77% 54.0M 1s 2024-09-21T23:50:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129800K .......... .......... .......... .......... .......... 77% 26.0M 1s 2024-09-21T23:50:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129850K .......... .......... .......... .......... .......... 77% 28.6M 1s 2024-09-21T23:50:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129900K .......... .......... .......... .......... .......... 77% 33.2M 1s 2024-09-21T23:50:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:129950K .......... .......... .......... .......... .......... 77% 19.8M 1s 2024-09-21T23:50:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130000K .......... .......... .......... .......... .......... 77% 25.6M 1s 2024-09-21T23:50:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130050K .......... .......... .......... .......... .......... 77% 43.8M 1s 2024-09-21T23:50:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130100K .......... .......... .......... .......... .......... 77% 25.3M 1s 2024-09-21T23:50:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130150K .......... .......... .......... .......... .......... 77% 36.2M 1s 2024-09-21T23:50:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130200K .......... .......... .......... .......... .......... 77% 24.2M 1s 2024-09-21T23:50:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130250K .......... .......... .......... .......... .......... 77% 48.5M 1s 2024-09-21T23:50:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130300K .......... .......... .......... .......... .......... 77% 24.4M 1s 2024-09-21T23:50:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130350K .......... .......... .......... .......... .......... 77% 24.3M 1s 2024-09-21T23:50:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130400K .......... .......... .......... .......... .......... 77% 28.2M 1s 2024-09-21T23:50:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130450K .......... .......... .......... .......... .......... 77% 29.5M 1s 2024-09-21T23:50:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130500K .......... .......... .......... .......... .......... 77% 27.9M 1s 2024-09-21T23:50:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130550K .......... .......... .......... .......... .......... 77% 38.0M 1s 2024-09-21T23:50:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130600K .......... .......... .......... .......... .......... 77% 24.2M 1s 2024-09-21T23:50:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130650K .......... .......... .......... .......... .......... 77% 28.6M 1s 2024-09-21T23:50:39.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130700K .......... .......... .......... .......... .......... 77% 31.9M 1s 2024-09-21T23:50:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130750K .......... .......... .......... .......... .......... 77% 23.9M 1s 2024-09-21T23:50:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130800K .......... .......... .......... .......... .......... 77% 46.8M 1s 2024-09-21T23:50:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130850K .......... .......... .......... .......... .......... 77% 25.0M 1s 2024-09-21T23:50:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130900K .......... .......... .......... .......... .......... 77% 36.5M 1s 2024-09-21T23:50:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:130950K .......... .......... .......... .......... .......... 77% 23.6M 1s 2024-09-21T23:50:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131000K .......... .......... .......... .......... .......... 77% 51.3M 1s 2024-09-21T23:50:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131050K .......... .......... .......... .......... .......... 77% 24.3M 1s 2024-09-21T23:50:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131100K .......... .......... .......... .......... .......... 77% 34.0M 1s 2024-09-21T23:50:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131150K .......... .......... .......... .......... .......... 77% 18.1M 1s 2024-09-21T23:50:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131200K .......... .......... .......... .......... .......... 77% 37.0M 1s 2024-09-21T23:50:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131250K .......... .......... .......... .......... .......... 77% 23.9M 1s 2024-09-21T23:50:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131300K .......... .......... .......... .......... .......... 77% 47.6M 1s 2024-09-21T23:50:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131350K .......... .......... .......... .......... .......... 77% 27.0M 1s 2024-09-21T23:50:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131400K .......... .......... .......... .......... .......... 77% 31.6M 1s 2024-09-21T23:50:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131450K .......... .......... .......... .......... .......... 78% 24.7M 1s 2024-09-21T23:50:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131500K .......... .......... .......... .......... .......... 78% 49.0M 1s 2024-09-21T23:50:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131550K .......... .......... .......... .......... .......... 78% 25.0M 1s 2024-09-21T23:50:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131600K .......... .......... .......... .......... .......... 78% 24.1M 1s 2024-09-21T23:50:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131650K .......... .......... .......... .......... .......... 78% 34.5M 1s 2024-09-21T23:50:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131700K .......... .......... .......... .......... .......... 78% 24.6M 1s 2024-09-21T23:50:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131750K .......... .......... .......... .......... .......... 78% 24.5M 1s 2024-09-21T23:50:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131800K .......... .......... .......... .......... .......... 78% 47.5M 1s 2024-09-21T23:50:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131850K .......... .......... .......... .......... .......... 78% 24.7M 1s 2024-09-21T23:50:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131900K .......... .......... .......... .......... .......... 78% 37.2M 1s 2024-09-21T23:50:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:131950K .......... .......... .......... .......... .......... 78% 23.1M 1s 2024-09-21T23:50:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132000K .......... .......... .......... .......... .......... 78% 25.6M 1s 2024-09-21T23:50:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132050K .......... .......... .......... .......... .......... 78% 48.3M 1s 2024-09-21T23:50:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132100K .......... .......... .......... .......... .......... 78% 27.7M 1s 2024-09-21T23:50:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132150K .......... .......... .......... .......... .......... 78% 31.5M 1s 2024-09-21T23:50:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132200K .......... .......... .......... .......... .......... 78% 23.2M 1s 2024-09-21T23:50:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132250K .......... .......... .......... .......... .......... 78% 23.9M 1s 2024-09-21T23:50:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132300K .......... .......... .......... .......... .......... 78% 55.8M 1s 2024-09-21T23:50:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132350K .......... .......... .......... .......... .......... 78% 23.2M 1s 2024-09-21T23:50:40.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132400K .......... .......... .......... .......... .......... 78% 28.1M 1s 2024-09-21T23:50:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132450K .......... .......... .......... .......... .......... 78% 31.4M 1s 2024-09-21T23:50:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132500K .......... .......... .......... .......... .......... 78% 25.0M 1s 2024-09-21T23:50:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132550K .......... .......... .......... .......... .......... 78% 44.4M 1s 2024-09-21T23:50:40.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132600K .......... .......... .......... .......... .......... 78% 23.1M 1s 2024-09-21T23:50:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132650K .......... .......... .......... .......... .......... 78% 39.9M 1s 2024-09-21T23:50:40.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132700K .......... .......... .......... .......... .......... 78% 25.2M 1s 2024-09-21T23:50:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132750K .......... .......... .......... .......... .......... 78% 23.9M 1s 2024-09-21T23:50:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132800K .......... .......... .......... .......... .......... 78% 22.7M 1s 2024-09-21T23:50:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132850K .......... .......... .......... .......... .......... 78% 55.8M 1s 2024-09-21T23:50:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132900K .......... .......... .......... .......... .......... 78% 23.4M 1s 2024-09-21T23:50:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:132950K .......... .......... .......... .......... .......... 78% 36.2M 1s 2024-09-21T23:50:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133000K .......... .......... .......... .......... .......... 78% 24.1M 1s 2024-09-21T23:50:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133050K .......... .......... .......... .......... .......... 78% 47.3M 1s 2024-09-21T23:50:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133100K .......... .......... .......... .......... .......... 78% 24.6M 1s 2024-09-21T23:50:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133150K .......... .......... .......... .......... .......... 79% 24.0M 1s 2024-09-21T23:50:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133200K .......... .......... .......... .......... .......... 79% 33.9M 1s 2024-09-21T23:50:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133250K .......... .......... .......... .......... .......... 79% 25.1M 1s 2024-09-21T23:50:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133300K .......... .......... .......... .......... .......... 79% 23.7M 1s 2024-09-21T23:50:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133350K .......... .......... .......... .......... .......... 79% 47.9M 1s 2024-09-21T23:50:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133400K .......... .......... .......... .......... .......... 79% 33.7M 1s 2024-09-21T23:50:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133450K .......... .......... .......... .......... .......... 79% 24.7M 1s 2024-09-21T23:50:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133500K .......... .......... .......... .......... .......... 79% 25.6M 1s 2024-09-21T23:50:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133550K .......... .......... .......... .......... .......... 79% 25.7M 1s 2024-09-21T23:50:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133600K .......... .......... .......... .......... .......... 79% 41.1M 1s 2024-09-21T23:50:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133650K .......... .......... .......... .......... .......... 79% 25.2M 1s 2024-09-21T23:50:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133700K .......... .......... .......... .......... .......... 79% 22.6M 1s 2024-09-21T23:50:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133750K .......... .......... .......... .......... .......... 79% 40.6M 1s 2024-09-21T23:50:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133800K .......... .......... .......... .......... .......... 79% 44.2M 1s 2024-09-21T23:50:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133850K .......... .......... .......... .......... .......... 79% 25.4M 1s 2024-09-21T23:50:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133900K .......... .......... .......... .......... .......... 79% 35.9M 1s 2024-09-21T23:50:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:133950K .......... .......... .......... .......... .......... 79% 23.3M 1s 2024-09-21T23:50:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134000K .......... .......... .......... .......... .......... 79% 24.5M 1s 2024-09-21T23:50:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134050K .......... .......... .......... .......... .......... 79% 27.2M 1s 2024-09-21T23:50:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134100K .......... .......... .......... .......... .......... 79% 43.4M 1s 2024-09-21T23:50:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134150K .......... .......... .......... .......... .......... 79% 24.2M 1s 2024-09-21T23:50:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134200K .......... .......... .......... .......... .......... 79% 36.9M 1s 2024-09-21T23:50:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134250K .......... .......... .......... .......... .......... 79% 25.5M 1s 2024-09-21T23:50:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134300K .......... .......... .......... .......... .......... 79% 40.8M 1s 2024-09-21T23:50:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134350K .......... .......... .......... .......... .......... 79% 26.3M 1s 2024-09-21T23:50:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134400K .......... .......... .......... .......... .......... 79% 22.9M 1s 2024-09-21T23:50:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134450K .......... .......... .......... .......... .......... 79% 23.4M 1s 2024-09-21T23:50:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134500K .......... .......... .......... .......... .......... 79% 37.4M 1s 2024-09-21T23:50:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134550K .......... .......... .......... .......... .......... 79% 23.9M 1s 2024-09-21T23:50:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134600K .......... .......... .......... .......... .......... 79% 47.6M 1s 2024-09-21T23:50:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134650K .......... .......... .......... .......... .......... 79% 25.0M 1s 2024-09-21T23:50:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134700K .......... .......... .......... .......... .......... 79% 36.1M 1s 2024-09-21T23:50:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134750K .......... .......... .......... .......... .......... 79% 24.1M 1s 2024-09-21T23:50:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134800K .......... .......... .......... .......... .......... 79% 24.9M 1s 2024-09-21T23:50:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134850K .......... .......... .......... .......... .......... 80% 48.0M 1s 2024-09-21T23:50:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134900K .......... .......... .......... .......... .......... 80% 24.9M 1s 2024-09-21T23:50:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:134950K .......... .......... .......... .......... .......... 80% 36.7M 1s 2024-09-21T23:50:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135000K .......... .......... .......... .......... .......... 80% 22.6M 1s 2024-09-21T23:50:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135050K .......... .......... .......... .......... .......... 80% 24.7M 1s 2024-09-21T23:50:40.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135100K .......... .......... .......... .......... .......... 80% 48.8M 1s 2024-09-21T23:50:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135150K .......... .......... .......... .......... .......... 80% 23.9M 1s 2024-09-21T23:50:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135200K .......... .......... .......... .......... .......... 80% 25.0M 1s 2024-09-21T23:50:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135250K .......... .......... .......... .......... .......... 80% 35.5M 1s 2024-09-21T23:50:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135300K .......... .......... .......... .......... .......... 80% 25.3M 1s 2024-09-21T23:50:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135350K .......... .......... .......... .......... .......... 80% 51.6M 1s 2024-09-21T23:50:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135400K .......... .......... .......... .......... .......... 80% 22.2M 1s 2024-09-21T23:50:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135450K .......... .......... .......... .......... .......... 80% 37.5M 1s 2024-09-21T23:50:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135500K .......... .......... .......... .......... .......... 80% 25.2M 1s 2024-09-21T23:50:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135550K .......... .......... .......... .......... .......... 80% 22.9M 1s 2024-09-21T23:50:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135600K .......... .......... .......... .......... .......... 80% 24.7M 1s 2024-09-21T23:50:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135650K .......... .......... .......... .......... .......... 80% 50.8M 1s 2024-09-21T23:50:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135700K .......... .......... .......... .......... .......... 80% 24.2M 1s 2024-09-21T23:50:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135750K .......... .......... .......... .......... .......... 80% 35.0M 1s 2024-09-21T23:50:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135800K .......... .......... .......... .......... .......... 80% 25.3M 1s 2024-09-21T23:50:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135850K .......... .......... .......... .......... .......... 80% 47.0M 1s 2024-09-21T23:50:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135900K .......... .......... .......... .......... .......... 80% 20.9M 1s 2024-09-21T23:50:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:135950K .......... .......... .......... .......... .......... 80% 29.1M 1s 2024-09-21T23:50:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136000K .......... .......... .......... .......... .......... 80% 38.1M 1s 2024-09-21T23:50:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136050K .......... .......... .......... .......... .......... 80% 24.3M 1s 2024-09-21T23:50:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136100K .......... .......... .......... .......... .......... 80% 23.3M 1s 2024-09-21T23:50:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136150K .......... .......... .......... .......... .......... 80% 49.0M 1s 2024-09-21T23:50:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136200K .......... .......... .......... .......... .......... 80% 35.3M 1s 2024-09-21T23:50:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136250K .......... .......... .......... .......... .......... 80% 24.6M 1s 2024-09-21T23:50:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136300K .......... .......... .......... .......... .......... 80% 24.4M 1s 2024-09-21T23:50:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136350K .......... .......... .......... .......... .......... 80% 24.6M 1s 2024-09-21T23:50:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136400K .......... .......... .......... .......... .......... 80% 46.3M 1s 2024-09-21T23:50:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136450K .......... .......... .......... .......... .......... 80% 25.6M 1s 2024-09-21T23:50:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136500K .......... .......... .......... .......... .......... 81% 35.6M 1s 2024-09-21T23:50:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136550K .......... .......... .......... .......... .......... 81% 24.1M 1s 2024-09-21T23:50:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136600K .......... .......... .......... .......... .......... 81% 50.0M 1s 2024-09-21T23:50:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136650K .......... .......... .......... .......... .......... 81% 27.2M 1s 2024-09-21T23:50:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136700K .......... .......... .......... .......... .......... 81% 30.0M 1s 2024-09-21T23:50:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136750K .......... .......... .......... .......... .......... 81% 23.4M 1s 2024-09-21T23:50:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136800K .......... .......... .......... .......... .......... 81% 24.9M 1s 2024-09-21T23:50:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136850K .......... .......... .......... .......... .......... 81% 25.1M 1s 2024-09-21T23:50:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136900K .......... .......... .......... .......... .......... 81% 55.0M 1s 2024-09-21T23:50:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:136950K .......... .......... .......... .......... .......... 81% 23.2M 1s 2024-09-21T23:50:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137000K .......... .......... .......... .......... .......... 81% 35.0M 1s 2024-09-21T23:50:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137050K .......... .......... .......... .......... .......... 81% 25.0M 1s 2024-09-21T23:50:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137100K .......... .......... .......... .......... .......... 81% 48.2M 1s 2024-09-21T23:50:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137150K .......... .......... .......... .......... .......... 81% 24.0M 1s 2024-09-21T23:50:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137200K .......... .......... .......... .......... .......... 81% 25.1M 1s 2024-09-21T23:50:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137250K .......... .......... .......... .......... .......... 81% 37.7M 1s 2024-09-21T23:50:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137300K .......... .......... .......... .......... .......... 81% 22.7M 1s 2024-09-21T23:50:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137350K .......... .......... .......... .......... .......... 81% 24.7M 1s 2024-09-21T23:50:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137400K .......... .......... .......... .......... .......... 81% 46.5M 1s 2024-09-21T23:50:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137450K .......... .......... .......... .......... .......... 81% 24.5M 1s 2024-09-21T23:50:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137500K .......... .......... .......... .......... .......... 81% 36.8M 1s 2024-09-21T23:50:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137550K .......... .......... .......... .......... .......... 81% 24.6M 1s 2024-09-21T23:50:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137600K .......... .......... .......... .......... .......... 81% 23.4M 1s 2024-09-21T23:50:40.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137650K .......... .......... .......... .......... .......... 81% 46.2M 1s 2024-09-21T23:50:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137700K .......... .......... .......... .......... .......... 81% 25.1M 1s 2024-09-21T23:50:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137750K .......... .......... .......... .......... .......... 81% 36.3M 1s 2024-09-21T23:50:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137800K .......... .......... .......... .......... .......... 81% 24.7M 1s 2024-09-21T23:50:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137850K .......... .......... .......... .......... .......... 81% 23.8M 1s 2024-09-21T23:50:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137900K .......... .......... .......... .......... .......... 81% 53.9M 1s 2024-09-21T23:50:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:137950K .......... .......... .......... .......... .......... 81% 24.6M 1s 2024-09-21T23:50:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138000K .......... .......... .......... .......... .......... 81% 22.8M 1s 2024-09-21T23:50:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138050K .......... .......... .......... .......... .......... 81% 37.4M 1s 2024-09-21T23:50:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138100K .......... .......... .......... .......... .......... 81% 25.0M 1s 2024-09-21T23:50:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138150K .......... .......... .......... .......... .......... 81% 47.5M 1s 2024-09-21T23:50:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138200K .......... .......... .......... .......... .......... 82% 23.3M 1s 2024-09-21T23:50:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138250K .......... .......... .......... .......... .......... 82% 39.4M 1s 2024-09-21T23:50:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138300K .......... .......... .......... .......... .......... 82% 24.6M 1s 2024-09-21T23:50:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138350K .......... .......... .......... .......... .......... 82% 23.9M 1s 2024-09-21T23:50:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138400K .......... .......... .......... .......... .......... 82% 23.4M 1s 2024-09-21T23:50:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138450K .......... .......... .......... .......... .......... 82% 51.9M 1s 2024-09-21T23:50:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138500K .......... .......... .......... .......... .......... 82% 23.7M 1s 2024-09-21T23:50:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138550K .......... .......... .......... .......... .......... 82% 37.6M 1s 2024-09-21T23:50:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138600K .......... .......... .......... .......... .......... 82% 23.6M 1s 2024-09-21T23:50:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138650K .......... .......... .......... .......... .......... 82% 47.0M 1s 2024-09-21T23:50:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138700K .......... .......... .......... .......... .......... 82% 24.9M 1s 2024-09-21T23:50:40.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138750K .......... .......... .......... .......... .......... 82% 24.2M 1s 2024-09-21T23:50:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138800K .......... .......... .......... .......... .......... 82% 36.0M 1s 2024-09-21T23:50:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138850K .......... .......... .......... .......... .......... 82% 25.3M 1s 2024-09-21T23:50:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138900K .......... .......... .......... .......... .......... 82% 48.2M 1s 2024-09-21T23:50:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:138950K .......... .......... .......... .......... .......... 82% 24.3M 1s 2024-09-21T23:50:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139000K .......... .......... .......... .......... .......... 82% 33.6M 1s 2024-09-21T23:50:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139050K .......... .......... .......... .......... .......... 82% 24.0M 1s 2024-09-21T23:50:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139100K .......... .......... .......... .......... .......... 82% 25.1M 1s 2024-09-21T23:50:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139150K .......... .......... .......... .......... .......... 82% 23.8M 1s 2024-09-21T23:50:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139200K .......... .......... .......... .......... .......... 82% 51.0M 1s 2024-09-21T23:50:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139250K .......... .......... .......... .......... .......... 82% 25.3M 1s 2024-09-21T23:50:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139300K .......... .......... .......... .......... .......... 82% 35.2M 1s 2024-09-21T23:50:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139350K .......... .......... .......... .......... .......... 82% 24.7M 1s 2024-09-21T23:50:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139400K .......... .......... .......... .......... .......... 82% 50.8M 1s 2024-09-21T23:50:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139450K .......... .......... .......... .......... .......... 82% 23.4M 1s 2024-09-21T23:50:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139500K .......... .......... .......... .......... .......... 82% 38.5M 1s 2024-09-21T23:50:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139550K .......... .......... .......... .......... .......... 82% 22.1M 1s 2024-09-21T23:50:40.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139600K .......... .......... .......... .......... .......... 82% 24.7M 1s 2024-09-21T23:50:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139650K .......... .......... .......... .......... .......... 82% 25.6M 1s 2024-09-21T23:50:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139700K .......... .......... .......... .......... .......... 82% 43.0M 1s 2024-09-21T23:50:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139750K .......... .......... .......... .......... .......... 82% 25.5M 1s 2024-09-21T23:50:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139800K .......... .......... .......... .......... .......... 82% 36.9M 1s 2024-09-21T23:50:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139850K .......... .......... .......... .......... .......... 82% 25.2M 1s 2024-09-21T23:50:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139900K .......... .......... .......... .......... .......... 83% 47.4M 1s 2024-09-21T23:50:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:139950K .......... .......... .......... .......... .......... 83% 23.6M 1s 2024-09-21T23:50:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140000K .......... .......... .......... .......... .......... 83% 24.8M 1s 2024-09-21T23:50:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140050K .......... .......... .......... .......... .......... 83% 37.8M 1s 2024-09-21T23:50:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140100K .......... .......... .......... .......... .......... 83% 23.6M 1s 2024-09-21T23:50:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140150K .......... .......... .......... .......... .......... 83% 24.7M 1s 2024-09-21T23:50:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140200K .......... .......... .......... .......... .......... 83% 50.2M 1s 2024-09-21T23:50:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140250K .......... .......... .......... .......... .......... 83% 22.8M 1s 2024-09-21T23:50:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140300K .......... .......... .......... .......... .......... 83% 39.5M 1s 2024-09-21T23:50:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140350K .......... .......... .......... .......... .......... 83% 23.2M 1s 2024-09-21T23:50:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140400K .......... .......... .......... .......... .......... 83% 25.7M 1s 2024-09-21T23:50:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140450K .......... .......... .......... .......... .......... 83% 43.3M 1s 2024-09-21T23:50:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140500K .......... .......... .......... .......... .......... 83% 26.0M 1s 2024-09-21T23:50:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140550K .......... .......... .......... .......... .......... 83% 36.0M 1s 2024-09-21T23:50:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140600K .......... .......... .......... .......... .......... 83% 24.7M 1s 2024-09-21T23:50:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140650K .......... .......... .......... .......... .......... 83% 24.1M 1s 2024-09-21T23:50:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140700K .......... .......... .......... .......... .......... 83% 46.2M 1s 2024-09-21T23:50:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140750K .......... .......... .......... .......... .......... 83% 24.8M 1s 2024-09-21T23:50:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140800K .......... .......... .......... .......... .......... 83% 22.4M 1s 2024-09-21T23:50:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140850K .......... .......... .......... .......... .......... 83% 54.6M 1s 2024-09-21T23:50:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140900K .......... .......... .......... .......... .......... 83% 21.7M 1s 2024-09-21T23:50:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:140950K .......... .......... .......... .......... .......... 83% 44.3M 1s 2024-09-21T23:50:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141000K .......... .......... .......... .......... .......... 83% 24.5M 1s 2024-09-21T23:50:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141050K .......... .......... .......... .......... .......... 83% 36.0M 1s 2024-09-21T23:50:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141100K .......... .......... .......... .......... .......... 83% 25.2M 1s 2024-09-21T23:50:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141150K .......... .......... .......... .......... .......... 83% 24.3M 1s 2024-09-21T23:50:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141200K .......... .......... .......... .......... .......... 83% 23.3M 1s 2024-09-21T23:50:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141250K .......... .......... .......... .......... .......... 83% 54.4M 1s 2024-09-21T23:50:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141300K .......... .......... .......... .......... .......... 83% 35.3M 1s 2024-09-21T23:50:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141350K .......... .......... .......... .......... .......... 83% 23.5M 1s 2024-09-21T23:50:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141400K .......... .......... .......... .......... .......... 83% 24.7M 1s 2024-09-21T23:50:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141450K .......... .......... .......... .......... .......... 83% 47.8M 1s 2024-09-21T23:50:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141500K .......... .......... .......... .......... .......... 83% 24.9M 1s 2024-09-21T23:50:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141550K .......... .......... .......... .......... .......... 84% 24.0M 1s 2024-09-21T23:50:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141600K .......... .......... .......... .......... .......... 84% 36.9M 1s 2024-09-21T23:50:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141650K .......... .......... .......... .......... .......... 84% 24.7M 1s 2024-09-21T23:50:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141700K .......... .......... .......... .......... .......... 84% 49.9M 1s 2024-09-21T23:50:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141750K .......... .......... .......... .......... .......... 84% 24.1M 1s 2024-09-21T23:50:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141800K .......... .......... .......... .......... .......... 84% 34.0M 1s 2024-09-21T23:50:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141850K .......... .......... .......... .......... .......... 84% 25.1M 1s 2024-09-21T23:50:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141900K .......... .......... .......... .......... .......... 84% 25.2M 1s 2024-09-21T23:50:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:141950K .......... .......... .......... .......... .......... 84% 26.1M 1s 2024-09-21T23:50:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142000K .......... .......... .......... .......... .......... 84% 39.3M 1s 2024-09-21T23:50:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142050K .......... .......... .......... .......... .......... 84% 24.6M 1s 2024-09-21T23:50:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142100K .......... .......... .......... .......... .......... 84% 36.9M 1s 2024-09-21T23:50:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142150K .......... .......... .......... .......... .......... 84% 42.1M 1s 2024-09-21T23:50:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142200K .......... .......... .......... .......... .......... 84% 29.5M 1s 2024-09-21T23:50:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142250K .......... .......... .......... .......... .......... 84% 22.4M 1s 2024-09-21T23:50:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142300K .......... .......... .......... .......... .......... 84% 26.5M 1s 2024-09-21T23:50:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142350K .......... .......... .......... .......... .......... 84% 28.6M 1s 2024-09-21T23:50:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142400K .......... .......... .......... .......... .......... 84% 25.6M 1s 2024-09-21T23:50:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142450K .......... .......... .......... .......... .......... 84% 50.8M 1s 2024-09-21T23:50:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142500K .......... .......... .......... .......... .......... 84% 22.8M 1s 2024-09-21T23:50:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142550K .......... .......... .......... .......... .......... 84% 27.5M 1s 2024-09-21T23:50:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142600K .......... .......... .......... .......... .......... 84% 35.5M 1s 2024-09-21T23:50:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142650K .......... .......... .......... .......... .......... 84% 24.9M 1s 2024-09-21T23:50:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142700K .......... .......... .......... .......... .......... 84% 46.6M 1s 2024-09-21T23:50:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142750K .......... .......... .......... .......... .......... 84% 18.1M 1s 2024-09-21T23:50:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142800K .......... .......... .......... .......... .......... 84% 36.3M 1s 2024-09-21T23:50:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142850K .......... .......... .......... .......... .......... 84% 37.8M 1s 2024-09-21T23:50:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142900K .......... .......... .......... .......... .......... 84% 25.0M 1s 2024-09-21T23:50:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:142950K .......... .......... .......... .......... .......... 84% 22.2M 1s 2024-09-21T23:50:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143000K .......... .......... .......... .......... .......... 84% 55.1M 1s 2024-09-21T23:50:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143050K .......... .......... .......... .......... .......... 84% 20.3M 1s 2024-09-21T23:50:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143100K .......... .......... .......... .......... .......... 84% 48.8M 1s 2024-09-21T23:50:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143150K .......... .......... .......... .......... .......... 84% 24.3M 1s 2024-09-21T23:50:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143200K .......... .......... .......... .......... .......... 84% 24.3M 1s 2024-09-21T23:50:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143250K .......... .......... .......... .......... .......... 85% 45.6M 1s 2024-09-21T23:50:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143300K .......... .......... .......... .......... .......... 85% 24.8M 1s 2024-09-21T23:50:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143350K .......... .......... .......... .......... .......... 85% 36.9M 1s 2024-09-21T23:50:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143400K .......... .......... .......... .......... .......... 85% 33.4M 1s 2024-09-21T23:50:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143450K .......... .......... .......... .......... .......... 85% 30.4M 1s 2024-09-21T23:50:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143500K .......... .......... .......... .......... .......... 85% 25.5M 1s 2024-09-21T23:50:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143550K .......... .......... .......... .......... .......... 85% 23.7M 1s 2024-09-21T23:50:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143600K .......... .......... .......... .......... .......... 85% 23.4M 1s 2024-09-21T23:50:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143650K .......... .......... .......... .......... .......... 85% 39.9M 1s 2024-09-21T23:50:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143700K .......... .......... .......... .......... .......... 85% 26.9M 1s 2024-09-21T23:50:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143750K .......... .......... .......... .......... .......... 85% 43.6M 1s 2024-09-21T23:50:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143800K .......... .......... .......... .......... .......... 85% 21.8M 1s 2024-09-21T23:50:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143850K .......... .......... .......... .......... .......... 85% 24.6M 1s 2024-09-21T23:50:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143900K .......... .......... .......... .......... .......... 85% 58.4M 1s 2024-09-21T23:50:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:143950K .......... .......... .......... .......... .......... 85% 19.4M 1s 2024-09-21T23:50:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144000K .......... .......... .......... .......... .......... 85% 46.7M 1s 2024-09-21T23:50:40.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144050K .......... .......... .......... .......... .......... 85% 25.0M 1s 2024-09-21T23:50:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144100K .......... .......... .......... .......... .......... 85% 34.4M 1s 2024-09-21T23:50:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144150K .......... .......... .......... .......... .......... 85% 22.7M 1s 2024-09-21T23:50:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144200K .......... .......... .......... .......... .......... 85% 24.3M 1s 2024-09-21T23:50:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144250K .......... .......... .......... .......... .......... 85% 51.8M 1s 2024-09-21T23:50:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144300K .......... .......... .......... .......... .......... 85% 24.8M 1s 2024-09-21T23:50:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144350K .......... .......... .......... .......... .......... 85% 21.5M 1s 2024-09-21T23:50:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144400K .......... .......... .......... .......... .......... 85% 45.1M 1s 2024-09-21T23:50:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144450K .......... .......... .......... .......... .......... 85% 25.3M 1s 2024-09-21T23:50:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144500K .......... .......... .......... .......... .......... 85% 48.2M 1s 2024-09-21T23:50:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144550K .......... .......... .......... .......... .......... 85% 24.9M 1s 2024-09-21T23:50:40.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144600K .......... .......... .......... .......... .......... 85% 22.6M 1s 2024-09-21T23:50:40.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144650K .......... .......... .......... .......... .......... 85% 38.1M 1s 2024-09-21T23:50:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144700K .......... .......... .......... .......... .......... 85% 23.7M 1s 2024-09-21T23:50:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144750K .......... .......... .......... .......... .......... 85% 23.2M 1s 2024-09-21T23:50:40.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144800K .......... .......... .......... .......... .......... 85% 51.5M 1s 2024-09-21T23:50:40.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144850K .......... .......... .......... .......... .......... 85% 25.4M 1s 2024-09-21T23:50:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144900K .......... .......... .......... .......... .......... 85% 34.3M 1s 2024-09-21T23:50:40.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:144950K .......... .......... .......... .......... .......... 86% 25.8M 1s 2024-09-21T23:50:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145000K .......... .......... .......... .......... .......... 86% 43.4M 1s 2024-09-21T23:50:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145050K .......... .......... .......... .......... .......... 86% 25.7M 1s 2024-09-21T23:50:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145100K .......... .......... .......... .......... .......... 86% 35.5M 1s 2024-09-21T23:50:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145150K .......... .......... .......... .......... .......... 86% 24.2M 1s 2024-09-21T23:50:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145200K .......... .......... .......... .......... .......... 86% 23.2M 1s 2024-09-21T23:50:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145250K .......... .......... .......... .......... .......... 86% 26.9M 1s 2024-09-21T23:50:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145300K .......... .......... .......... .......... .......... 86% 55.8M 1s 2024-09-21T23:50:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145350K .......... .......... .......... .......... .......... 86% 22.6M 1s 2024-09-21T23:50:40.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145400K .......... .......... .......... .......... .......... 86% 32.2M 1s 2024-09-21T23:50:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145450K .......... .......... .......... .......... .......... 86% 22.1M 1s 2024-09-21T23:50:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145500K .......... .......... .......... .......... .......... 86% 58.7M 1s 2024-09-21T23:50:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145550K .......... .......... .......... .......... .......... 86% 24.3M 1s 2024-09-21T23:50:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145600K .......... .......... .......... .......... .......... 86% 24.5M 1s 2024-09-21T23:50:40.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145650K .......... .......... .......... .......... .......... 86% 40.9M 1s 2024-09-21T23:50:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145700K .......... .......... .......... .......... .......... 86% 22.8M 1s 2024-09-21T23:50:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145750K .......... .......... .......... .......... .......... 86% 23.0M 1s 2024-09-21T23:50:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145800K .......... .......... .......... .......... .......... 86% 55.0M 1s 2024-09-21T23:50:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145850K .......... .......... .......... .......... .......... 86% 33.5M 1s 2024-09-21T23:50:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145900K .......... .......... .......... .......... .......... 86% 25.4M 1s 2024-09-21T23:50:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:145950K .......... .......... .......... .......... .......... 86% 25.3M 1s 2024-09-21T23:50:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146000K .......... .......... .......... .......... .......... 86% 24.2M 1s 2024-09-21T23:50:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146050K .......... .......... .......... .......... .......... 86% 43.6M 1s 2024-09-21T23:50:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146100K .......... .......... .......... .......... .......... 86% 25.9M 1s 2024-09-21T23:50:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146150K .......... .......... .......... .......... .......... 86% 37.2M 1s 2024-09-21T23:50:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146200K .......... .......... .......... .......... .......... 86% 23.5M 1s 2024-09-21T23:50:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146250K .......... .......... .......... .......... .......... 86% 50.0M 1s 2024-09-21T23:50:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146300K .......... .......... .......... .......... .......... 86% 24.0M 1s 2024-09-21T23:50:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146350K .......... .......... .......... .......... .......... 86% 23.8M 1s 2024-09-21T23:50:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146400K .......... .......... .......... .......... .......... 86% 29.7M 1s 2024-09-21T23:50:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146450K .......... .......... .......... .......... .......... 86% 23.8M 1s 2024-09-21T23:50:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146500K .......... .......... .......... .......... .......... 86% 24.1M 1s 2024-09-21T23:50:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146550K .......... .......... .......... .......... .......... 86% 42.7M 1s 2024-09-21T23:50:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146600K .......... .......... .......... .......... .......... 86% 26.2M 1s 2024-09-21T23:50:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146650K .......... .......... .......... .......... .......... 87% 36.6M 1s 2024-09-21T23:50:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146700K .......... .......... .......... .......... .......... 87% 25.2M 1s 2024-09-21T23:50:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146750K .......... .......... .......... .......... .......... 87% 25.0M 1s 2024-09-21T23:50:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146800K .......... .......... .......... .......... .......... 87% 41.3M 1s 2024-09-21T23:50:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146850K .......... .......... .......... .......... .......... 87% 24.9M 1s 2024-09-21T23:50:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146900K .......... .......... .......... .......... .......... 87% 35.0M 1s 2024-09-21T23:50:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:146950K .......... .......... .......... .......... .......... 87% 23.4M 1s 2024-09-21T23:50:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147000K .......... .......... .......... .......... .......... 87% 25.7M 1s 2024-09-21T23:50:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147050K .......... .......... .......... .......... .......... 87% 48.7M 1s 2024-09-21T23:50:40.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147100K .......... .......... .......... .......... .......... 87% 24.9M 1s 2024-09-21T23:50:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147150K .......... .......... .......... .......... .......... 87% 23.9M 1s 2024-09-21T23:50:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147200K .......... .......... .......... .......... .......... 87% 36.0M 1s 2024-09-21T23:50:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147250K .......... .......... .......... .......... .......... 87% 24.8M 1s 2024-09-21T23:50:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147300K .......... .......... .......... .......... .......... 87% 44.7M 1s 2024-09-21T23:50:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147350K .......... .......... .......... .......... .......... 87% 24.8M 1s 2024-09-21T23:50:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147400K .......... .......... .......... .......... .......... 87% 37.7M 1s 2024-09-21T23:50:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147450K .......... .......... .......... .......... .......... 87% 23.0M 1s 2024-09-21T23:50:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147500K .......... .......... .......... .......... .......... 87% 25.8M 1s 2024-09-21T23:50:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147550K .......... .......... .......... .......... .......... 87% 24.1M 1s 2024-09-21T23:50:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147600K .......... .......... .......... .......... .......... 87% 45.8M 1s 2024-09-21T23:50:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147650K .......... .......... .......... .......... .......... 87% 24.6M 1s 2024-09-21T23:50:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147700K .......... .......... .......... .......... .......... 87% 34.9M 1s 2024-09-21T23:50:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147750K .......... .......... .......... .......... .......... 87% 25.7M 1s 2024-09-21T23:50:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147800K .......... .......... .......... .......... .......... 87% 46.5M 1s 2024-09-21T23:50:40.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147850K .......... .......... .......... .......... .......... 87% 25.1M 1s 2024-09-21T23:50:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147900K .......... .......... .......... .......... .......... 87% 35.5M 1s 2024-09-21T23:50:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:147950K .......... .......... .......... .......... .......... 87% 24.1M 1s 2024-09-21T23:50:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148000K .......... .......... .......... .......... .......... 87% 22.8M 1s 2024-09-21T23:50:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148050K .......... .......... .......... .......... .......... 87% 25.1M 1s 2024-09-21T23:50:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148100K .......... .......... .......... .......... .......... 87% 57.0M 1s 2024-09-21T23:50:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148150K .......... .......... .......... .......... .......... 87% 31.4M 1s 2024-09-21T23:50:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148200K .......... .......... .......... .......... .......... 87% 25.3M 1s 2024-09-21T23:50:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148250K .......... .......... .......... .......... .......... 87% 24.7M 1s 2024-09-21T23:50:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148300K .......... .......... .......... .......... .......... 88% 66.6M 1s 2024-09-21T23:50:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148350K .......... .......... .......... .......... .......... 88% 22.7M 1s 2024-09-21T23:50:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148400K .......... .......... .......... .......... .......... 88% 30.6M 1s 2024-09-21T23:50:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148450K .......... .......... .......... .......... .......... 88% 26.8M 1s 2024-09-21T23:50:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148500K .......... .......... .......... .......... .......... 88% 23.0M 1s 2024-09-21T23:50:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148550K .......... .......... .......... .......... .......... 88% 45.7M 1s 2024-09-21T23:50:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148600K .......... .......... .......... .......... .......... 88% 25.6M 1s 2024-09-21T23:50:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148650K .......... .......... .......... .......... .......... 88% 33.4M 1s 2024-09-21T23:50:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148700K .......... .......... .......... .......... .......... 88% 25.6M 1s 2024-09-21T23:50:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148750K .......... .......... .......... .......... .......... 88% 24.5M 1s 2024-09-21T23:50:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148800K .......... .......... .......... .......... .......... 88% 24.4M 1s 2024-09-21T23:50:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148850K .......... .......... .......... .......... .......... 88% 46.0M 1s 2024-09-21T23:50:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148900K .......... .......... .......... .......... .......... 88% 25.3M 1s 2024-09-21T23:50:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:148950K .......... .......... .......... .......... .......... 88% 36.1M 1s 2024-09-21T23:50:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149000K .......... .......... .......... .......... .......... 88% 25.4M 1s 2024-09-21T23:50:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149050K .......... .......... .......... .......... .......... 88% 52.5M 1s 2024-09-21T23:50:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149100K .......... .......... .......... .......... .......... 88% 23.9M 1s 2024-09-21T23:50:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149150K .......... .......... .......... .......... .......... 88% 22.8M 1s 2024-09-21T23:50:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149200K .......... .......... .......... .......... .......... 88% 34.3M 1s 2024-09-21T23:50:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149250K .......... .......... .......... .......... .......... 88% 24.6M 1s 2024-09-21T23:50:40.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149300K .......... .......... .......... .......... .......... 88% 25.3M 1s 2024-09-21T23:50:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149350K .......... .......... .......... .......... .......... 88% 44.0M 1s 2024-09-21T23:50:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149400K .......... .......... .......... .......... .......... 88% 27.2M 1s 2024-09-21T23:50:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149450K .......... .......... .......... .......... .......... 88% 32.7M 1s 2024-09-21T23:50:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149500K .......... .......... .......... .......... .......... 88% 24.7M 1s 2024-09-21T23:50:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149550K .......... .......... .......... .......... .......... 88% 25.0M 1s 2024-09-21T23:50:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149600K .......... .......... .......... .......... .......... 88% 45.7M 1s 2024-09-21T23:50:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149650K .......... .......... .......... .......... .......... 88% 24.4M 1s 2024-09-21T23:50:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149700K .......... .......... .......... .......... .......... 88% 27.9M 1s 2024-09-21T23:50:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149750K .......... .......... .......... .......... .......... 88% 29.4M 1s 2024-09-21T23:50:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149800K .......... .......... .......... .......... .......... 88% 24.6M 1s 2024-09-21T23:50:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149850K .......... .......... .......... .......... .......... 88% 45.1M 1s 2024-09-21T23:50:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149900K .......... .......... .......... .......... .......... 88% 25.6M 1s 2024-09-21T23:50:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:149950K .......... .......... .......... .......... .......... 88% 23.2M 1s 2024-09-21T23:50:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150000K .......... .......... .......... .......... .......... 89% 38.3M 1s 2024-09-21T23:50:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150050K .......... .......... .......... .......... .......... 89% 24.8M 1s 2024-09-21T23:50:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150100K .......... .......... .......... .......... .......... 89% 46.9M 1s 2024-09-21T23:50:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150150K .......... .......... .......... .......... .......... 89% 23.1M 1s 2024-09-21T23:50:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150200K .......... .......... .......... .......... .......... 89% 39.0M 1s 2024-09-21T23:50:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150250K .......... .......... .......... .......... .......... 89% 24.8M 1s 2024-09-21T23:50:40.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150300K .......... .......... .......... .......... .......... 89% 23.4M 1s 2024-09-21T23:50:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150350K .......... .......... .......... .......... .......... 89% 23.5M 1s 2024-09-21T23:50:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150400K .......... .......... .......... .......... .......... 89% 45.9M 1s 2024-09-21T23:50:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150450K .......... .......... .......... .......... .......... 89% 25.0M 1s 2024-09-21T23:50:40.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150500K .......... .......... .......... .......... .......... 89% 37.1M 1s 2024-09-21T23:50:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150550K .......... .......... .......... .......... .......... 89% 24.3M 1s 2024-09-21T23:50:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150600K .......... .......... .......... .......... .......... 89% 47.1M 1s 2024-09-21T23:50:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150650K .......... .......... .......... .......... .......... 89% 24.8M 1s 2024-09-21T23:50:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150700K .......... .......... .......... .......... .......... 89% 35.7M 1s 2024-09-21T23:50:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150750K .......... .......... .......... .......... .......... 89% 24.0M 1s 2024-09-21T23:50:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150800K .......... .......... .......... .......... .......... 89% 25.1M 1s 2024-09-21T23:50:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150850K .......... .......... .......... .......... .......... 89% 23.2M 1s 2024-09-21T23:50:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150900K .......... .......... .......... .......... .......... 89% 56.6M 1s 2024-09-21T23:50:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:150950K .......... .......... .......... .......... .......... 89% 31.6M 1s 2024-09-21T23:50:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151000K .......... .......... .......... .......... .......... 89% 24.9M 1s 2024-09-21T23:50:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151050K .......... .......... .......... .......... .......... 89% 24.3M 1s 2024-09-21T23:50:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151100K .......... .......... .......... .......... .......... 89% 54.5M 1s 2024-09-21T23:50:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151150K .......... .......... .......... .......... .......... 89% 22.2M 1s 2024-09-21T23:50:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151200K .......... .......... .......... .......... .......... 89% 25.7M 1s 2024-09-21T23:50:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151250K .......... .......... .......... .......... .......... 89% 40.6M 1s 2024-09-21T23:50:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151300K .......... .......... .......... .......... .......... 89% 23.1M 1s 2024-09-21T23:50:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151350K .......... .......... .......... .......... .......... 89% 44.0M 1s 2024-09-21T23:50:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151400K .......... .......... .......... .......... .......... 89% 24.9M 1s 2024-09-21T23:50:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151450K .......... .......... .......... .......... .......... 89% 34.2M 1s 2024-09-21T23:50:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151500K .......... .......... .......... .......... .......... 89% 25.6M 1s 2024-09-21T23:50:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151550K .......... .......... .......... .......... .......... 89% 23.5M 1s 2024-09-21T23:50:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151600K .......... .......... .......... .......... .......... 89% 23.9M 1s 2024-09-21T23:50:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151650K .......... .......... .......... .......... .......... 89% 53.0M 1s 2024-09-21T23:50:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151700K .......... .......... .......... .......... .......... 90% 25.2M 1s 2024-09-21T23:50:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151750K .......... .......... .......... .......... .......... 90% 33.8M 1s 2024-09-21T23:50:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151800K .......... .......... .......... .......... .......... 90% 25.4M 1s 2024-09-21T23:50:40.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151850K .......... .......... .......... .......... .......... 90% 44.7M 1s 2024-09-21T23:50:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151900K .......... .......... .......... .......... .......... 90% 25.3M 1s 2024-09-21T23:50:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:151950K .......... .......... .......... .......... .......... 90% 23.9M 1s 2024-09-21T23:50:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152000K .......... .......... .......... .......... .......... 90% 32.9M 1s 2024-09-21T23:50:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152050K .......... .......... .......... .......... .......... 90% 24.4M 1s 2024-09-21T23:50:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152100K .......... .......... .......... .......... .......... 90% 24.2M 1s 2024-09-21T23:50:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152150K .......... .......... .......... .......... .......... 90% 51.9M 1s 2024-09-21T23:50:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152200K .......... .......... .......... .......... .......... 90% 24.3M 1s 2024-09-21T23:50:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152250K .......... .......... .......... .......... .......... 90% 36.4M 1s 2024-09-21T23:50:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152300K .......... .......... .......... .......... .......... 90% 25.1M 1s 2024-09-21T23:50:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152350K .......... .......... .......... .......... .......... 90% 24.5M 1s 2024-09-21T23:50:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152400K .......... .......... .......... .......... .......... 90% 45.9M 1s 2024-09-21T23:50:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152450K .......... .......... .......... .......... .......... 90% 25.7M 1s 2024-09-21T23:50:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152500K .......... .......... .......... .......... .......... 90% 33.8M 1s 2024-09-21T23:50:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152550K .......... .......... .......... .......... .......... 90% 23.6M 1s 2024-09-21T23:50:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152600K .......... .......... .......... .......... .......... 90% 24.6M 1s 2024-09-21T23:50:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152650K .......... .......... .......... .......... .......... 90% 55.2M 1s 2024-09-21T23:50:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152700K .......... .......... .......... .......... .......... 90% 24.1M 1s 2024-09-21T23:50:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152750K .......... .......... .......... .......... .......... 90% 23.4M 1s 2024-09-21T23:50:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152800K .......... .......... .......... .......... .......... 90% 37.0M 1s 2024-09-21T23:50:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152850K .......... .......... .......... .......... .......... 90% 24.8M 1s 2024-09-21T23:50:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152900K .......... .......... .......... .......... .......... 90% 48.8M 1s 2024-09-21T23:50:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:152950K .......... .......... .......... .......... .......... 90% 24.2M 1s 2024-09-21T23:50:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153000K .......... .......... .......... .......... .......... 90% 35.3M 1s 2024-09-21T23:50:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153050K .......... .......... .......... .......... .......... 90% 24.4M 1s 2024-09-21T23:50:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153100K .......... .......... .......... .......... .......... 90% 34.2M 1s 2024-09-21T23:50:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153150K .......... .......... .......... .......... .......... 90% 23.4M 1s 2024-09-21T23:50:40.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153200K .......... .......... .......... .......... .......... 90% 35.9M 1s 2024-09-21T23:50:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153250K .......... .......... .......... .......... .......... 90% 26.9M 1s 2024-09-21T23:50:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153300K .......... .......... .......... .......... .......... 90% 30.3M 1s 2024-09-21T23:50:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153350K .......... .......... .......... .......... .......... 91% 42.3M 1s 2024-09-21T23:50:40.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153400K .......... .......... .......... .......... .......... 91% 27.8M 1s 2024-09-21T23:50:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153450K .......... .......... .......... .......... .......... 91% 22.3M 1s 2024-09-21T23:50:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153500K .......... .......... .......... .......... .......... 91% 44.1M 1s 2024-09-21T23:50:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153550K .......... .......... .......... .......... .......... 91% 22.5M 1s 2024-09-21T23:50:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153600K .......... .......... .......... .......... .......... 91% 23.9M 1s 2024-09-21T23:50:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153650K .......... .......... .......... .......... .......... 91% 28.2M 1s 2024-09-21T23:50:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153700K .......... .......... .......... .......... .......... 91% 37.7M 1s 2024-09-21T23:50:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153750K .......... .......... .......... .......... .......... 91% 23.9M 1s 2024-09-21T23:50:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153800K .......... .......... .......... .......... .......... 91% 36.3M 1s 2024-09-21T23:50:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153850K .......... .......... .......... .......... .......... 91% 27.5M 1s 2024-09-21T23:50:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153900K .......... .......... .......... .......... .......... 91% 38.5M 1s 2024-09-21T23:50:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:153950K .......... .......... .......... .......... .......... 91% 24.5M 1s 2024-09-21T23:50:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154000K .......... .......... .......... .......... .......... 91% 25.3M 0s 2024-09-21T23:50:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154050K .......... .......... .......... .......... .......... 91% 35.8M 0s 2024-09-21T23:50:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154100K .......... .......... .......... .......... .......... 91% 25.3M 0s 2024-09-21T23:50:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154150K .......... .......... .......... .......... .......... 91% 47.7M 0s 2024-09-21T23:50:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154200K .......... .......... .......... .......... .......... 91% 23.1M 0s 2024-09-21T23:50:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154250K .......... .......... .......... .......... .......... 91% 38.8M 0s 2024-09-21T23:50:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154300K .......... .......... .......... .......... .......... 91% 22.2M 0s 2024-09-21T23:50:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154350K .......... .......... .......... .......... .......... 91% 22.8M 0s 2024-09-21T23:50:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154400K .......... .......... .......... .......... .......... 91% 26.3M 0s 2024-09-21T23:50:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154450K .......... .......... .......... .......... .......... 91% 49.1M 0s 2024-09-21T23:50:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154500K .......... .......... .......... .......... .......... 91% 32.4M 0s 2024-09-21T23:50:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154550K .......... .......... .......... .......... .......... 91% 26.8M 0s 2024-09-21T23:50:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154600K .......... .......... .......... .......... .......... 91% 24.4M 0s 2024-09-21T23:50:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154650K .......... .......... .......... .......... .......... 91% 49.0M 0s 2024-09-21T23:50:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154700K .......... .......... .......... .......... .......... 91% 25.0M 0s 2024-09-21T23:50:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154750K .......... .......... .......... .......... .......... 91% 23.9M 0s 2024-09-21T23:50:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154800K .......... .......... .......... .......... .......... 91% 37.2M 0s 2024-09-21T23:50:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154850K .......... .......... .......... .......... .......... 91% 23.6M 0s 2024-09-21T23:50:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154900K .......... .......... .......... .......... .......... 91% 24.6M 0s 2024-09-21T23:50:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:154950K .......... .......... .......... .......... .......... 91% 48.1M 0s 2024-09-21T23:50:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155000K .......... .......... .......... .......... .......... 91% 23.9M 0s 2024-09-21T23:50:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155050K .......... .......... .......... .......... .......... 92% 39.7M 0s 2024-09-21T23:50:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155100K .......... .......... .......... .......... .......... 92% 23.8M 0s 2024-09-21T23:50:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155150K .......... .......... .......... .......... .......... 92% 24.2M 0s 2024-09-21T23:50:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155200K .......... .......... .......... .......... .......... 92% 46.9M 0s 2024-09-21T23:50:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155250K .......... .......... .......... .......... .......... 92% 25.0M 0s 2024-09-21T23:50:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155300K .......... .......... .......... .......... .......... 92% 33.9M 0s 2024-09-21T23:50:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155350K .......... .......... .......... .......... .......... 92% 33.0M 0s 2024-09-21T23:50:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155400K .......... .......... .......... .......... .......... 92% 23.5M 0s 2024-09-21T23:50:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155450K .......... .......... .......... .......... .......... 92% 34.6M 0s 2024-09-21T23:50:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155500K .......... .......... .......... .......... .......... 92% 32.6M 0s 2024-09-21T23:50:40.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155550K .......... .......... .......... .......... .......... 92% 18.3M 0s 2024-09-21T23:50:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155600K .......... .......... .......... .......... .......... 92% 37.3M 0s 2024-09-21T23:50:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155650K .......... .......... .......... .......... .......... 92% 24.6M 0s 2024-09-21T23:50:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155700K .......... .......... .......... .......... .......... 92% 47.6M 0s 2024-09-21T23:50:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155750K .......... .......... .......... .......... .......... 92% 25.0M 0s 2024-09-21T23:50:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155800K .......... .......... .......... .......... .......... 92% 35.0M 0s 2024-09-21T23:50:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155850K .......... .......... .......... .......... .......... 92% 26.0M 0s 2024-09-21T23:50:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155900K .......... .......... .......... .......... .......... 92% 43.0M 0s 2024-09-21T23:50:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:155950K .......... .......... .......... .......... .......... 92% 16.5M 0s 2024-09-21T23:50:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156000K .......... .......... .......... .......... .......... 92% 47.8M 0s 2024-09-21T23:50:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156050K .......... .......... .......... .......... .......... 92% 22.8M 0s 2024-09-21T23:50:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156100K .......... .......... .......... .......... .......... 92% 40.0M 0s 2024-09-21T23:50:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156150K .......... .......... .......... .......... .......... 92% 24.1M 0s 2024-09-21T23:50:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156200K .......... .......... .......... .......... .......... 92% 43.2M 0s 2024-09-21T23:50:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156250K .......... .......... .......... .......... .......... 92% 25.1M 0s 2024-09-21T23:50:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156300K .......... .......... .......... .......... .......... 92% 38.5M 0s 2024-09-21T23:50:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156350K .......... .......... .......... .......... .......... 92% 24.2M 0s 2024-09-21T23:50:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156400K .......... .......... .......... .......... .......... 92% 23.6M 0s 2024-09-21T23:50:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156450K .......... .......... .......... .......... .......... 92% 47.4M 0s 2024-09-21T23:50:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156500K .......... .......... .......... .......... .......... 92% 25.2M 0s 2024-09-21T23:50:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156550K .......... .......... .......... .......... .......... 92% 34.2M 0s 2024-09-21T23:50:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156600K .......... .......... .......... .......... .......... 92% 23.8M 0s 2024-09-21T23:50:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156650K .......... .......... .......... .......... .......... 92% 23.9M 0s 2024-09-21T23:50:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156700K .......... .......... .......... .......... .......... 92% 55.6M 0s 2024-09-21T23:50:40.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156750K .......... .......... .......... .......... .......... 93% 23.8M 0s 2024-09-21T23:50:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156800K .......... .......... .......... .......... .......... 93% 24.0M 0s 2024-09-21T23:50:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156850K .......... .......... .......... .......... .......... 93% 27.5M 0s 2024-09-21T23:50:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156900K .......... .......... .......... .......... .......... 93% 33.8M 0s 2024-09-21T23:50:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:156950K .......... .......... .......... .......... .......... 93% 47.0M 0s 2024-09-21T23:50:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157000K .......... .......... .......... .......... .......... 93% 24.2M 0s 2024-09-21T23:50:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157050K .......... .......... .......... .......... .......... 93% 39.1M 0s 2024-09-21T23:50:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157100K .......... .......... .......... .......... .......... 93% 21.4M 0s 2024-09-21T23:50:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157150K .......... .......... .......... .......... .......... 93% 24.4M 0s 2024-09-21T23:50:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157200K .......... .......... .......... .......... .......... 93% 25.8M 0s 2024-09-21T23:50:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157250K .......... .......... .......... .......... .......... 93% 44.6M 0s 2024-09-21T23:50:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157300K .......... .......... .......... .......... .......... 93% 25.7M 0s 2024-09-21T23:50:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157350K .......... .......... .......... .......... .......... 93% 34.2M 0s 2024-09-21T23:50:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157400K .......... .......... .......... .......... .......... 93% 25.2M 0s 2024-09-21T23:50:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157450K .......... .......... .......... .......... .......... 93% 44.4M 0s 2024-09-21T23:50:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157500K .......... .......... .......... .......... .......... 93% 25.9M 0s 2024-09-21T23:50:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157550K .......... .......... .......... .......... .......... 93% 25.0M 0s 2024-09-21T23:50:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157600K .......... .......... .......... .......... .......... 93% 33.8M 0s 2024-09-21T23:50:40.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157650K .......... .......... .......... .......... .......... 93% 23.0M 0s 2024-09-21T23:50:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157700K .......... .......... .......... .......... .......... 93% 10.7M 0s 2024-09-21T23:50:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157750K .......... .......... .......... .......... .......... 93% 68.1M 0s 2024-09-21T23:50:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157800K .......... .......... .......... .......... .......... 93% 80.6M 0s 2024-09-21T23:50:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157850K .......... .......... .......... .......... .......... 93% 80.2M 0s 2024-09-21T23:50:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157900K .......... .......... .......... .......... .......... 93% 29.0M 0s 2024-09-21T23:50:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:157950K .......... .......... .......... .......... .......... 93% 23.3M 0s 2024-09-21T23:50:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158000K .......... .......... .......... .......... .......... 93% 44.6M 0s 2024-09-21T23:50:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158050K .......... .......... .......... .......... .......... 93% 24.9M 0s 2024-09-21T23:50:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158100K .......... .......... .......... .......... .......... 93% 34.2M 0s 2024-09-21T23:50:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158150K .......... .......... .......... .......... .......... 93% 27.8M 0s 2024-09-21T23:50:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158200K .......... .......... .......... .......... .......... 93% 22.3M 0s 2024-09-21T23:50:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158250K .......... .......... .......... .......... .......... 93% 57.0M 0s 2024-09-21T23:50:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158300K .......... .......... .......... .......... .......... 93% 22.6M 0s 2024-09-21T23:50:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158350K .......... .......... .......... .......... .......... 93% 23.0M 0s 2024-09-21T23:50:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158400K .......... .......... .......... .......... .......... 93% 38.3M 0s 2024-09-21T23:50:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158450K .......... .......... .......... .......... .......... 94% 25.4M 0s 2024-09-21T23:50:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158500K .......... .......... .......... .......... .......... 94% 44.8M 0s 2024-09-21T23:50:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158550K .......... .......... .......... .......... .......... 94% 24.5M 0s 2024-09-21T23:50:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158600K .......... .......... .......... .......... .......... 94% 36.5M 0s 2024-09-21T23:50:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158650K .......... .......... .......... .......... .......... 94% 26.1M 0s 2024-09-21T23:50:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158700K .......... .......... .......... .......... .......... 94% 43.1M 0s 2024-09-21T23:50:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158750K .......... .......... .......... .......... .......... 94% 16.1M 0s 2024-09-21T23:50:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158800K .......... .......... .......... .......... .......... 94% 59.9M 0s 2024-09-21T23:50:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158850K .......... .......... .......... .......... .......... 94% 29.2M 0s 2024-09-21T23:50:40.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158900K .......... .......... .......... .......... .......... 94% 24.6M 0s 2024-09-21T23:50:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:158950K .......... .......... .......... .......... .......... 94% 23.9M 0s 2024-09-21T23:50:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159000K .......... .......... .......... .......... .......... 94% 50.0M 0s 2024-09-21T23:50:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159050K .......... .......... .......... .......... .......... 94% 26.3M 0s 2024-09-21T23:50:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159100K .......... .......... .......... .......... .......... 94% 32.3M 0s 2024-09-21T23:50:40.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159150K .......... .......... .......... .......... .......... 94% 25.7M 0s 2024-09-21T23:50:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159200K .......... .......... .......... .......... .......... 94% 25.1M 0s 2024-09-21T23:50:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159250K .......... .......... .......... .......... .......... 94% 47.9M 0s 2024-09-21T23:50:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159300K .......... .......... .......... .......... .......... 94% 25.1M 0s 2024-09-21T23:50:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159350K .......... .......... .......... .......... .......... 94% 29.8M 0s 2024-09-21T23:50:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159400K .......... .......... .......... .......... .......... 94% 26.1M 0s 2024-09-21T23:50:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159450K .......... .......... .......... .......... .......... 94% 26.1M 0s 2024-09-21T23:50:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159500K .......... .......... .......... .......... .......... 94% 42.3M 0s 2024-09-21T23:50:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159550K .......... .......... .......... .......... .......... 94% 23.7M 0s 2024-09-21T23:50:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159600K .......... .......... .......... .......... .......... 94% 25.7M 0s 2024-09-21T23:50:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159650K .......... .......... .......... .......... .......... 94% 35.4M 0s 2024-09-21T23:50:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159700K .......... .......... .......... .......... .......... 94% 27.4M 0s 2024-09-21T23:50:40.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159750K .......... .......... .......... .......... .......... 94% 48.0M 0s 2024-09-21T23:50:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159800K .......... .......... .......... .......... .......... 94% 22.6M 0s 2024-09-21T23:50:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159850K .......... .......... .......... .......... .......... 94% 35.6M 0s 2024-09-21T23:50:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159900K .......... .......... .......... .......... .......... 94% 26.0M 0s 2024-09-21T23:50:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:159950K .......... .......... .......... .......... .......... 94% 23.8M 0s 2024-09-21T23:50:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160000K .......... .......... .......... .......... .......... 94% 22.8M 0s 2024-09-21T23:50:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160050K .......... .......... .......... .......... .......... 94% 45.9M 0s 2024-09-21T23:50:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160100K .......... .......... .......... .......... .......... 95% 24.8M 0s 2024-09-21T23:50:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160150K .......... .......... .......... .......... .......... 95% 34.9M 0s 2024-09-21T23:50:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160200K .......... .......... .......... .......... .......... 95% 24.5M 0s 2024-09-21T23:50:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160250K .......... .......... .......... .......... .......... 95% 46.3M 0s 2024-09-21T23:50:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160300K .......... .......... .......... .......... .......... 95% 25.2M 0s 2024-09-21T23:50:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160350K .......... .......... .......... .......... .......... 95% 24.7M 0s 2024-09-21T23:50:40.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160400K .......... .......... .......... .......... .......... 95% 34.6M 0s 2024-09-21T23:50:40.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160450K .......... .......... .......... .......... .......... 95% 26.2M 0s 2024-09-21T23:50:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160500K .......... .......... .......... .......... .......... 95% 23.2M 0s 2024-09-21T23:50:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160550K .......... .......... .......... .......... .......... 95% 52.0M 0s 2024-09-21T23:50:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160600K .......... .......... .......... .......... .......... 95% 23.5M 0s 2024-09-21T23:50:40.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160650K .......... .......... .......... .......... .......... 95% 34.7M 0s 2024-09-21T23:50:40.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160700K .......... .......... .......... .......... .......... 95% 24.8M 0s 2024-09-21T23:50:40.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160750K .......... .......... .......... .......... .......... 95% 34.1M 0s 2024-09-21T23:50:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160800K .......... .......... .......... .......... .......... 95% 27.0M 0s 2024-09-21T23:50:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160850K .......... .......... .......... .......... .......... 95% 26.8M 0s 2024-09-21T23:50:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160900K .......... .......... .......... .......... .......... 95% 36.1M 0s 2024-09-21T23:50:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:160950K .......... .......... .......... .......... .......... 95% 24.9M 0s 2024-09-21T23:50:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161000K .......... .......... .......... .......... .......... 95% 42.0M 0s 2024-09-21T23:50:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161050K .......... .......... .......... .......... .......... 95% 27.3M 0s 2024-09-21T23:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161100K .......... .......... .......... .......... .......... 95% 30.6M 0s 2024-09-21T23:50:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161150K .......... .......... .......... .......... .......... 95% 23.9M 0s 2024-09-21T23:50:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161200K .......... .......... .......... .......... .......... 95% 25.8M 0s 2024-09-21T23:50:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161250K .......... .......... .......... .......... .......... 95% 26.2M 0s 2024-09-21T23:50:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161300K .......... .......... .......... .......... .......... 95% 43.7M 0s 2024-09-21T23:50:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161350K .......... .......... .......... .......... .......... 95% 24.9M 0s 2024-09-21T23:50:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161400K .......... .......... .......... .......... .......... 95% 34.9M 0s 2024-09-21T23:50:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161450K .......... .......... .......... .......... .......... 95% 26.8M 0s 2024-09-21T23:50:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161500K .......... .......... .......... .......... .......... 95% 50.6M 0s 2024-09-21T23:50:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161550K .......... .......... .......... .......... .......... 95% 24.2M 0s 2024-09-21T23:50:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161600K .......... .......... .......... .......... .......... 95% 23.3M 0s 2024-09-21T23:50:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161650K .......... .......... .......... .......... .......... 95% 28.1M 0s 2024-09-21T23:50:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161700K .......... .......... .......... .......... .......... 95% 25.5M 0s 2024-09-21T23:50:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161750K .......... .......... .......... .......... .......... 95% 25.0M 0s 2024-09-21T23:50:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161800K .......... .......... .......... .......... .......... 96% 45.7M 0s 2024-09-21T23:50:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161850K .......... .......... .......... .......... .......... 96% 25.8M 0s 2024-09-21T23:50:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161900K .......... .......... .......... .......... .......... 96% 33.0M 0s 2024-09-21T23:50:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:161950K .......... .......... .......... .......... .......... 96% 25.9M 0s 2024-09-21T23:50:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162000K .......... .......... .......... .......... .......... 96% 24.6M 0s 2024-09-21T23:50:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162050K .......... .......... .......... .......... .......... 96% 46.3M 0s 2024-09-21T23:50:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162100K .......... .......... .......... .......... .......... 96% 25.8M 0s 2024-09-21T23:50:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162150K .......... .......... .......... .......... .......... 96% 34.8M 0s 2024-09-21T23:50:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162200K .......... .......... .......... .......... .......... 96% 23.2M 0s 2024-09-21T23:50:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162250K .......... .......... .......... .......... .......... 96% 24.1M 0s 2024-09-21T23:50:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162300K .......... .......... .......... .......... .......... 96% 61.6M 0s 2024-09-21T23:50:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162350K .......... .......... .......... .......... .......... 96% 24.7M 0s 2024-09-21T23:50:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162400K .......... .......... .......... .......... .......... 96% 28.7M 0s 2024-09-21T23:50:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162450K .......... .......... .......... .......... .......... 96% 26.7M 0s 2024-09-21T23:50:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162500K .......... .......... .......... .......... .......... 96% 30.8M 0s 2024-09-21T23:50:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162550K .......... .......... .......... .......... .......... 96% 40.2M 0s 2024-09-21T23:50:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162600K .......... .......... .......... .......... .......... 96% 22.7M 0s 2024-09-21T23:50:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162650K .......... .......... .......... .......... .......... 96% 34.8M 0s 2024-09-21T23:50:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162700K .......... .......... .......... .......... .......... 96% 26.0M 0s 2024-09-21T23:50:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162750K .......... .......... .......... .......... .......... 96% 21.6M 0s 2024-09-21T23:50:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162800K .......... .......... .......... .......... .......... 96% 28.6M 0s 2024-09-21T23:50:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162850K .......... .......... .......... .......... .......... 96% 37.5M 0s 2024-09-21T23:50:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162900K .......... .......... .......... .......... .......... 96% 24.9M 0s 2024-09-21T23:50:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:162950K .......... .......... .......... .......... .......... 96% 34.2M 0s 2024-09-21T23:50:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163000K .......... .......... .......... .......... .......... 96% 26.4M 0s 2024-09-21T23:50:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163050K .......... .......... .......... .......... .......... 96% 41.5M 0s 2024-09-21T23:50:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163100K .......... .......... .......... .......... .......... 96% 27.0M 0s 2024-09-21T23:50:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163150K .......... .......... .......... .......... .......... 96% 23.4M 0s 2024-09-21T23:50:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163200K .......... .......... .......... .......... .......... 96% 37.5M 0s 2024-09-21T23:50:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163250K .......... .......... .......... .......... .......... 96% 25.5M 0s 2024-09-21T23:50:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163300K .......... .......... .......... .......... .......... 96% 22.8M 0s 2024-09-21T23:50:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163350K .......... .......... .......... .......... .......... 96% 50.5M 0s 2024-09-21T23:50:41.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163400K .......... .......... .......... .......... .......... 96% 22.7M 0s 2024-09-21T23:50:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163450K .......... .......... .......... .......... .......... 96% 37.9M 0s 2024-09-21T23:50:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163500K .......... .......... .......... .......... .......... 97% 25.2M 0s 2024-09-21T23:50:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163550K .......... .......... .......... .......... .......... 97% 24.2M 0s 2024-09-21T23:50:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163600K .......... .......... .......... .......... .......... 97% 45.0M 0s 2024-09-21T23:50:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163650K .......... .......... .......... .......... .......... 97% 26.0M 0s 2024-09-21T23:50:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163700K .......... .......... .......... .......... .......... 97% 35.9M 0s 2024-09-21T23:50:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163750K .......... .......... .......... .......... .......... 97% 25.2M 0s 2024-09-21T23:50:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163800K .......... .......... .......... .......... .......... 97% 41.2M 0s 2024-09-21T23:50:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163850K .......... .......... .......... .......... .......... 97% 26.0M 0s 2024-09-21T23:50:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163900K .......... .......... .......... .......... .......... 97% 29.6M 0s 2024-09-21T23:50:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:163950K .......... .......... .......... .......... .......... 97% 21.8M 0s 2024-09-21T23:50:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164000K .......... .......... .......... .......... .......... 97% 27.4M 0s 2024-09-21T23:50:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164050K .......... .......... .......... .......... .......... 97% 25.2M 0s 2024-09-21T23:50:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164100K .......... .......... .......... .......... .......... 97% 44.9M 0s 2024-09-21T23:50:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164150K .......... .......... .......... .......... .......... 97% 25.0M 0s 2024-09-21T23:50:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164200K .......... .......... .......... .......... .......... 97% 37.7M 0s 2024-09-21T23:50:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164250K .......... .......... .......... .......... .......... 97% 24.0M 0s 2024-09-21T23:50:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164300K .......... .......... .......... .......... .......... 97% 47.9M 0s 2024-09-21T23:50:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164350K .......... .......... .......... .......... .......... 97% 23.9M 0s 2024-09-21T23:50:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164400K .......... .......... .......... .......... .......... 97% 25.4M 0s 2024-09-21T23:50:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164450K .......... .......... .......... .......... .......... 97% 33.5M 0s 2024-09-21T23:50:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164500K .......... .......... .......... .......... .......... 97% 24.6M 0s 2024-09-21T23:50:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164550K .......... .......... .......... .......... .......... 97% 24.9M 0s 2024-09-21T23:50:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164600K .......... .......... .......... .......... .......... 97% 45.5M 0s 2024-09-21T23:50:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164650K .......... .......... .......... .......... .......... 97% 24.7M 0s 2024-09-21T23:50:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164700K .......... .......... .......... .......... .......... 97% 37.7M 0s 2024-09-21T23:50:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164750K .......... .......... .......... .......... .......... 97% 23.6M 0s 2024-09-21T23:50:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164800K .......... .......... .......... .......... .......... 97% 24.7M 0s 2024-09-21T23:50:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164850K .......... .......... .......... .......... .......... 97% 47.8M 0s 2024-09-21T23:50:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164900K .......... .......... .......... .......... .......... 97% 25.5M 0s 2024-09-21T23:50:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:164950K .......... .......... .......... .......... .......... 97% 37.0M 0s 2024-09-21T23:50:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165000K .......... .......... .......... .......... .......... 97% 22.7M 0s 2024-09-21T23:50:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165050K .......... .......... .......... .......... .......... 97% 24.3M 0s 2024-09-21T23:50:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165100K .......... .......... .......... .......... .......... 97% 56.1M 0s 2024-09-21T23:50:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165150K .......... .......... .......... .......... .......... 98% 22.7M 0s 2024-09-21T23:50:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165200K .......... .......... .......... .......... .......... 98% 25.0M 0s 2024-09-21T23:50:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165250K .......... .......... .......... .......... .......... 98% 36.2M 0s 2024-09-21T23:50:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165300K .......... .......... .......... .......... .......... 98% 25.2M 0s 2024-09-21T23:50:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165350K .......... .......... .......... .......... .......... 98% 45.2M 0s 2024-09-21T23:50:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165400K .......... .......... .......... .......... .......... 98% 27.7M 0s 2024-09-21T23:50:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165450K .......... .......... .......... .......... .......... 98% 30.3M 0s 2024-09-21T23:50:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165500K .......... .......... .......... .......... .......... 98% 25.3M 0s 2024-09-21T23:50:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165550K .......... .......... .......... .......... .......... 98% 22.2M 0s 2024-09-21T23:50:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165600K .......... .......... .......... .......... .......... 98% 26.1M 0s 2024-09-21T23:50:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165650K .......... .......... .......... .......... .......... 98% 58.9M 0s 2024-09-21T23:50:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165700K .......... .......... .......... .......... .......... 98% 22.2M 0s 2024-09-21T23:50:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165750K .......... .......... .......... .......... .......... 98% 33.7M 0s 2024-09-21T23:50:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165800K .......... .......... .......... .......... .......... 98% 25.0M 0s 2024-09-21T23:50:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165850K .......... .......... .......... .......... .......... 98% 47.9M 0s 2024-09-21T23:50:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165900K .......... .......... .......... .......... .......... 98% 25.3M 0s 2024-09-21T23:50:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:165950K .......... .......... .......... .......... .......... 98% 24.3M 0s 2024-09-21T23:50:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166000K .......... .......... .......... .......... .......... 98% 36.1M 0s 2024-09-21T23:50:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166050K .......... .......... .......... .......... .......... 98% 25.3M 0s 2024-09-21T23:50:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166100K .......... .......... .......... .......... .......... 98% 45.4M 0s 2024-09-21T23:50:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166150K .......... .......... .......... .......... .......... 98% 24.8M 0s 2024-09-21T23:50:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166200K .......... .......... .......... .......... .......... 98% 30.8M 0s 2024-09-21T23:50:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166250K .......... .......... .......... .......... .......... 98% 25.7M 0s 2024-09-21T23:50:41.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166300K .......... .......... .......... .......... .......... 98% 41.2M 0s 2024-09-21T23:50:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166350K .......... .......... .......... .......... .......... 98% 18.1M 0s 2024-09-21T23:50:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166400K .......... .......... .......... .......... .......... 98% 45.2M 0s 2024-09-21T23:50:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166450K .......... .......... .......... .......... .......... 98% 25.3M 0s 2024-09-21T23:50:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166500K .......... .......... .......... .......... .......... 98% 24.6M 0s 2024-09-21T23:50:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166550K .......... .......... .......... .......... .......... 98% 38.9M 0s 2024-09-21T23:50:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166600K .......... .......... .......... .......... .......... 98% 42.8M 0s 2024-09-21T23:50:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166650K .......... .......... .......... .......... .......... 98% 24.2M 0s 2024-09-21T23:50:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166700K .......... .......... .......... .......... .......... 98% 37.1M 0s 2024-09-21T23:50:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166750K .......... .......... .......... .......... .......... 98% 22.2M 0s 2024-09-21T23:50:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166800K .......... .......... .......... .......... .......... 98% 21.3M 0s 2024-09-21T23:50:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166850K .......... .......... .......... .......... .......... 99% 30.2M 0s 2024-09-21T23:50:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166900K .......... .......... .......... .......... .......... 99% 51.0M 0s 2024-09-21T23:50:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:166950K .......... .......... .......... .......... .......... 99% 23.7M 0s 2024-09-21T23:50:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167000K .......... .......... .......... .......... .......... 99% 32.7M 0s 2024-09-21T23:50:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167050K .......... .......... .......... .......... .......... 99% 26.2M 0s 2024-09-21T23:50:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167100K .......... .......... .......... .......... .......... 99% 44.7M 0s 2024-09-21T23:50:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167150K .......... .......... .......... .......... .......... 99% 24.0M 0s 2024-09-21T23:50:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167200K .......... .......... .......... .......... .......... 99% 25.6M 0s 2024-09-21T23:50:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167250K .......... .......... .......... .......... .......... 99% 35.7M 0s 2024-09-21T23:50:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167300K .......... .......... .......... .......... .......... 99% 22.8M 0s 2024-09-21T23:50:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167350K .......... .......... .......... .......... .......... 99% 24.5M 0s 2024-09-21T23:50:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167400K .......... .......... .......... .......... .......... 99% 59.5M 0s 2024-09-21T23:50:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167450K .......... .......... .......... .......... .......... 99% 22.2M 0s 2024-09-21T23:50:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167500K .......... .......... .......... .......... .......... 99% 35.6M 0s 2024-09-21T23:50:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167550K .......... .......... .......... .......... .......... 99% 28.6M 0s 2024-09-21T23:50:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167600K .......... .......... .......... .......... .......... 99% 25.8M 0s 2024-09-21T23:50:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167650K .......... .......... .......... .......... .......... 99% 35.3M 0s 2024-09-21T23:50:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167700K .......... .......... .......... .......... .......... 99% 25.8M 0s 2024-09-21T23:50:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167750K .......... .......... .......... .......... .......... 99% 39.6M 0s 2024-09-21T23:50:41.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167800K .......... .......... .......... .......... .......... 99% 22.9M 0s 2024-09-21T23:50:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167850K .......... .......... .......... .......... .......... 99% 23.1M 0s 2024-09-21T23:50:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167900K .......... .......... .......... .......... .......... 99% 52.5M 0s 2024-09-21T23:50:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:167950K .......... .......... .......... .......... .......... 99% 24.2M 0s 2024-09-21T23:50:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:168000K .......... .......... .......... .......... .......... 99% 24.7M 0s 2024-09-21T23:50:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:168050K .......... .......... .......... .......... .......... 99% 32.0M 0s 2024-09-21T23:50:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:168100K .......... .......... .......... .......... .......... 99% 25.3M 0s 2024-09-21T23:50:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:168150K .......... .......... .......... .......... .......... 99% 42.4M 0s 2024-09-21T23:50:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:168200K .......... .......... .......... .......... .......... 99% 26.6M 0s 2024-09-21T23:50:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:168250K .......... .......... .......... .......... .......... 99% 34.7M 0s 2024-09-21T23:50:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:168300K .......... .......... .......... .......... .......... 99% 26.1M 0s 2024-09-21T23:50:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:168350K .......... .......... .......... .......... .......... 99% 24.4M 0s 2024-09-21T23:50:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:168400K .......... .......... .......... .......... .......... 99% 24.5M 0s 2024-09-21T23:50:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:168450K .......... .......... .......... .......... .......... 99% 38.6M 0s 2024-09-21T23:50:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:168500K .......... .......... .......... .......... .......... 99% 35.2M 0s 2024-09-21T23:50:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:168550K .......... .......... . 100% 71.3M=5.8s 2024-09-21T23:50:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-09-21T23:50:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:2024-09-21 23:50:41 (28.5 MB/s) - 'linux-5.4.tar.gz' saved [172616875/172616875] 2024-09-21T23:50:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: 2024-09-21T23:50:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..10} 2024-09-21T23:50:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS=() 2024-09-21T23:50:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS=() 2024-09-21T23:50:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS=() 2024-09-21T23:50:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS=() 2024-09-21T23:50:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ RET=0 2024-09-21T23:50:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:50:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-21T23:50:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary1 2024-09-21T23:50:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-21T23:50:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:50:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:50:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-21T23:50:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:50:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:50:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:50:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:50:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-21T23:50:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:50:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:50:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:50:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-21T23:50:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:50:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-21T23:50:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:50:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-21T23:50:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary1': 2024-09-21T23:50:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:50:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:50:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:50:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10b38c14a135 2024-09-21T23:50:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10b38c14a135 2024-09-21T23:50:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:50:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:50:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:50:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:50:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:50:41 2024 2024-09-21T23:50:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:50:41 2024 2024-09-21T23:50:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:50:41 2024 2024-09-21T23:50:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:50:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:50:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 5a998568-0063-4af3-8f52-846373db4249 2024-09-21T23:50:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:50:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:50:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:50:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-21T23:50:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd0 2024-09-21T23:50:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:50:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-21T23:50:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:50:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-21T23:50:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:50:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: e7f7f7cf-0935-4676-924e-089dab33fceb 2024-09-21T23:50:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:50:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:50:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:50:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:50:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:50:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:50:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary1 2024-09-21T23:50:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-21T23:50:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:50:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary1 2024-09-21T23:50:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:50:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-21T23:50:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:50:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:50:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:50:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-21T23:50:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary2 2024-09-21T23:50:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary1 2024-09-21T23:50:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-21T23:50:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:50:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:50:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:50:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:50:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:50:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary1 2024-09-21T23:50:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-21T23:50:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-21T23:50:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:50:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:50:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:50:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:50:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-21T23:50:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:50:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:50:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:50:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-21T23:50:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:50:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-21T23:50:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:50:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-21T23:50:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:50:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 4 2024-09-21T23:50:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-21T23:50:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary2': 2024-09-21T23:50:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:50:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:50:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:50:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10c226996c80 2024-09-21T23:50:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10c226996c80 2024-09-21T23:50:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:50:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:50:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:50:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:50:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:50:42 2024 2024-09-21T23:50:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:50:42 2024 2024-09-21T23:50:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:50:42 2024 2024-09-21T23:50:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:50:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:50:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 13a89cd3-4094-4d4b-aa75-2fadc75b5a12 2024-09-21T23:50:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:50:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:50:42.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:50:42.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-21T23:50:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd1 2024-09-21T23:50:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:50:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-21T23:50:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:50:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-21T23:50:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:50:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: da52aa69-825e-45d1-a798-14441574bbcd 2024-09-21T23:50:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:50:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:50:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:50:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:50:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:50:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:50:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary2 2024-09-21T23:50:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-21T23:50:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:50:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary2 2024-09-21T23:50:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:50:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-21T23:50:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:50:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:50:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:50:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:50:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:50:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary2 2024-09-21T23:50:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:50:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-21T23:50:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary3 2024-09-21T23:50:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-21T23:50:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary2 2024-09-21T23:50:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-21T23:50:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-21T23:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:50:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:50:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:50:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-21T23:50:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:50:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:50:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:50:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-21T23:50:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:50:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-21T23:50:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:50:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-21T23:50:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:50:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 6 2024-09-21T23:50:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-21T23:50:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary3': 2024-09-21T23:50:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:50:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:50:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:50:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10d1a76b33ca 2024-09-21T23:50:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10d1a76b33ca 2024-09-21T23:50:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:50:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:50:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:50:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:50:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:50:43 2024 2024-09-21T23:50:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:50:43 2024 2024-09-21T23:50:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:50:43 2024 2024-09-21T23:50:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:50:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:50:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 839cdadb-f02d-4927-8770-46a1da127cb6 2024-09-21T23:50:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:50:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:50:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:50:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-21T23:50:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd2 2024-09-21T23:50:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:50:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-21T23:50:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:50:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-21T23:50:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:50:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: bca991c4-0df4-41ea-8360-7983a0638e1c 2024-09-21T23:50:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:50:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:50:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:50:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:50:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:50:44.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:50:44.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary3 2024-09-21T23:50:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-21T23:50:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:50:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary3 2024-09-21T23:50:44.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:50:44.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-21T23:50:44.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:44.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:50:44.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:50:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:50:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-21T23:50:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:50:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary4 2024-09-21T23:50:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary3 2024-09-21T23:50:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-21T23:50:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:50:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:50:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:50:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary3 2024-09-21T23:50:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-21T23:50:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:50:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-21T23:50:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:50:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:50:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:50:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:50:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-21T23:50:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:50:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:50:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:50:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-21T23:50:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:50:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-21T23:50:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:50:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-21T23:50:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:50:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 8 2024-09-21T23:50:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-21T23:50:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary4': 2024-09-21T23:50:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:50:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:50:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:50:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10e0b644bf9c 2024-09-21T23:50:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10e0b644bf9c 2024-09-21T23:50:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:50:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:50:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:50:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:50:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:50:44 2024 2024-09-21T23:50:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:50:44 2024 2024-09-21T23:50:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:50:44 2024 2024-09-21T23:50:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:50:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:50:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 840fe61d-df40-46a1-b554-3a9c492652e1 2024-09-21T23:50:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:50:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:50:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:50:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-21T23:50:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd3 2024-09-21T23:50:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:50:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-21T23:50:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:50:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-21T23:50:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:50:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 718b3e9a-5745-4e19-a725-d22086b99ea6 2024-09-21T23:50:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:50:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:50:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:50:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:50:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:50:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:50:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary4 2024-09-21T23:50:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-21T23:50:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:50:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary4 2024-09-21T23:50:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:50:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-21T23:50:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:50:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:50:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-21T23:50:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary5 2024-09-21T23:50:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-21T23:50:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary4 2024-09-21T23:50:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:50:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:50:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-21T23:50:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary4 2024-09-21T23:50:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:50:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:50:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-21T23:50:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:50:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:50:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:50:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-21T23:50:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:50:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:50:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:50:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:50:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:50:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:50:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-21T23:50:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:50:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-21T23:50:45.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:50:45.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-21T23:50:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:50:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:50:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:50:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:50:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 10 2024-09-21T23:50:45.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-21T23:50:45.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary5': 2024-09-21T23:50:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:50:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:50:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:50:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 10ef77ada42f 2024-09-21T23:50:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.10ef77ada42f 2024-09-21T23:50:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:50:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:50:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:50:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:50:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:50:45 2024 2024-09-21T23:50:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:50:45 2024 2024-09-21T23:50:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:50:45 2024 2024-09-21T23:50:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:50:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:50:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: afd18b9a-3089-4aac-8192-22348f467f8e 2024-09-21T23:50:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:50:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:50:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:50:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-21T23:50:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd4 2024-09-21T23:50:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:50:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-21T23:50:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:50:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-21T23:50:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:50:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 040cdac0-b804-48e4-a05e-69754a777ba8 2024-09-21T23:50:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:50:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:50:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:50:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:50:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:50:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:50:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:46.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary5 2024-09-21T23:50:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-21T23:50:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:50:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary5 2024-09-21T23:50:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:50:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-21T23:50:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:50:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:50:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:50:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:50:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:50:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:50:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-21T23:50:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary6 2024-09-21T23:50:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary5 2024-09-21T23:50:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-21T23:50:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:50:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary5 2024-09-21T23:50:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:50:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-21T23:50:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-21T23:50:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:50:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:50:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:50:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:50:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-21T23:50:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:50:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:50:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:50:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-21T23:50:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:50:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-21T23:50:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:50:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-21T23:50:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:50:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:50:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:50:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:50:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-21T23:50:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 13 2024-09-21T23:50:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary6': 2024-09-21T23:50:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:50:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:50:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:50:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11018a920dea 2024-09-21T23:50:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11018a920dea 2024-09-21T23:50:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:50:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:50:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:50:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:50:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:50:46 2024 2024-09-21T23:50:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:50:46 2024 2024-09-21T23:50:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:50:46 2024 2024-09-21T23:50:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:50:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:50:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: f4db6039-7ed4-4bde-a08f-e5c3a52196dd 2024-09-21T23:50:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:50:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:50:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:50:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-21T23:50:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd5 2024-09-21T23:50:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:50:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-21T23:50:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:50:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-21T23:50:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:50:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 03cc70c1-058e-46ca-90e4-bee8d9834668 2024-09-21T23:50:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:50:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:50:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:50:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:50:46.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:50:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:50:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary6 2024-09-21T23:50:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-21T23:50:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:50:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary6 2024-09-21T23:50:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:50:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-21T23:50:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:50:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:50:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:50:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:50:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:50:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:50:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary6 2024-09-21T23:50:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-21T23:50:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary7 2024-09-21T23:50:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-21T23:50:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary6 2024-09-21T23:50:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-21T23:50:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:50:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:50:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-21T23:50:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:50:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:50:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:50:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:50:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-21T23:50:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:50:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:50:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:50:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-21T23:50:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:50:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-21T23:50:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:50:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-21T23:50:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 12 2024-09-21T23:50:47.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:47.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:50:47.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:47.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:47.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:50:47.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:50:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:50:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-21T23:50:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 16 2024-09-21T23:50:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary7': 2024-09-21T23:50:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:50:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:50:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:50:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11121494247a 2024-09-21T23:50:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11121494247a 2024-09-21T23:50:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:50:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:50:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:50:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:50:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:50:47 2024 2024-09-21T23:50:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:50:47 2024 2024-09-21T23:50:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:50:47 2024 2024-09-21T23:50:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:50:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:50:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 0c8e1f25-bb70-411d-ae70-cb8bd563725c 2024-09-21T23:50:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:50:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:50:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:50:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-21T23:50:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd6 2024-09-21T23:50:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:50:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-21T23:50:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:50:47.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-21T23:50:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:50:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: baddd29e-b1c3-4c04-9337-53c723ba54e2 2024-09-21T23:50:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:50:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:50:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:50:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:50:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:50:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:50:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary7 2024-09-21T23:50:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-21T23:50:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:50:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary7 2024-09-21T23:50:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:50:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-21T23:50:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:50:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:50:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:50:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:50:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:50:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:50:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-21T23:50:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary8 2024-09-21T23:50:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary7 2024-09-21T23:50:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-21T23:50:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:50:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:50:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary7 2024-09-21T23:50:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-21T23:50:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-21T23:50:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:50:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:50:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:50:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:50:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-21T23:50:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:50:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:50:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:50:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-21T23:50:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:50:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-21T23:50:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:50:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-21T23:50:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:50:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:50:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:50:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:50:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-21T23:50:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary8': 2024-09-21T23:50:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:50:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:50:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:50:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 112329970384 2024-09-21T23:50:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.112329970384 2024-09-21T23:50:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:50:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:50:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:50:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:50:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:50:48 2024 2024-09-21T23:50:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:50:48 2024 2024-09-21T23:50:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:50:48 2024 2024-09-21T23:50:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:50:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:50:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 6ae0ec6a-c621-4b56-b1bb-8c919d2e9972 2024-09-21T23:50:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:50:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:50:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:50:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-21T23:50:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 19 2024-09-21T23:50:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:48.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd7 2024-09-21T23:50:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:50:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-21T23:50:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:50:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 524288/2621440 done 2024-09-21T23:50:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:50:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 259adacc-c6a7-4914-990a-61d651895cdb 2024-09-21T23:50:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:50:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:50:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:50:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:50:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:50:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:50:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary8 2024-09-21T23:50:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-21T23:50:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:50:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary8 2024-09-21T23:50:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:50:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-21T23:50:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:50:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:50:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:50:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-21T23:50:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary9 2024-09-21T23:50:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:50:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-21T23:50:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary8 2024-09-21T23:50:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:50:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:50:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:50:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:50:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary8 2024-09-21T23:50:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-21T23:50:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-21T23:50:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:50:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:50:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:50:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:50:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-21T23:50:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:50:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:50:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:50:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-21T23:50:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:50:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-21T23:50:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:50:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-21T23:50:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 15 2024-09-21T23:50:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 18 2024-09-21T23:50:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:50:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:50:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:50:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:50:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 22 2024-09-21T23:50:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-21T23:50:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary9': 2024-09-21T23:50:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:50:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:50:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:50:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11341bda727d 2024-09-21T23:50:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11341bda727d 2024-09-21T23:50:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:50:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:50:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:50:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:50:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:50:49 2024 2024-09-21T23:50:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:50:49 2024 2024-09-21T23:50:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:50:49 2024 2024-09-21T23:50:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:50:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:50:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: caa38a51-cb46-43c6-b6d7-895eff46ab74 2024-09-21T23:50:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:50:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:50:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:50:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-21T23:50:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd8 2024-09-21T23:50:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:50:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-21T23:50:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:50:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-21T23:50:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:50:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 88c558bc-2f39-4b97-813f-50ab325c50a9 2024-09-21T23:50:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:50:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:50:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:50:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:50:50.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:50:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:50:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary9 2024-09-21T23:50:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-21T23:50:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:50:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary9 2024-09-21T23:50:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:50:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-21T23:50:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:50:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:50:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:50:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:50:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:50:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary9 2024-09-21T23:50:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:50:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-21T23:50:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary10 2024-09-21T23:50:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-21T23:50:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary9 2024-09-21T23:50:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-21T23:50:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:50:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:50:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-21T23:50:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:50:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:50:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:50:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:50:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:50:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-21T23:50:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:50:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:50:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:50:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-21T23:50:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:50:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:50:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-21T23:50:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:50:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-21T23:50:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:50:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:50:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:50:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 21 2024-09-21T23:50:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:50:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 25 2024-09-21T23:50:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-21T23:50:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:50:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:50:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:50:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary10': 2024-09-21T23:50:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:50:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:50:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:50:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 11459456fac3 2024-09-21T23:50:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.11459456fac3 2024-09-21T23:50:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:50:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:50:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:50:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:50:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:50:50 2024 2024-09-21T23:50:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:50:50 2024 2024-09-21T23:50:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:50:50 2024 2024-09-21T23:50:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:50:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:50:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 6697eeff-3345-479b-b250-0bb39fb9675c 2024-09-21T23:50:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:50:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:50:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:50:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-21T23:50:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd9 2024-09-21T23:50:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:50:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-21T23:50:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:50:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-21T23:50:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:50:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 14f339e7-6026-48df-a45c-0995573d28af 2024-09-21T23:50:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:50:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:50:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:50:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:50:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:50:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:50:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:50:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary10 2024-09-21T23:50:51.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-21T23:50:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:50:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary10 2024-09-21T23:50:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:50:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-21T23:50:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:50:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:50:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:50:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:50:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:50:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary10 2024-09-21T23:50:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-21T23:50:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 30600 2024-09-21T23:50:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary10 2024-09-21T23:50:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-21T23:50:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:50:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-21T23:50:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 24 2024-09-21T23:50:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 29 2024-09-21T23:50:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:50:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:50:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:50:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:50:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:50:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:50:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 27 2024-09-21T23:50:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:50:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:50:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:50:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:50:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:50:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:50:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 28 2024-09-21T23:50:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 30 2024-09-21T23:50:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 32 2024-09-21T23:50:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:50:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:50:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:50:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:50:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:50:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:50:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:50:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:50:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:50:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 33 2024-09-21T23:50:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:50:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:50:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:50:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:50:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:50:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:50:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 34 2024-09-21T23:50:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 31 2024-09-21T23:50:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:50:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:50:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:50:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:50:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:50:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:50:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:50:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:50:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:50:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 35 2024-09-21T23:50:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 36 2024-09-21T23:50:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 37 2024-09-21T23:50:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 38 2024-09-21T23:50:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:50:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:50:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:50:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 39 2024-09-21T23:50:57.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 40 2024-09-21T23:50:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 41 2024-09-21T23:50:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:50:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:50:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:50:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:50:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:50:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:50:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 42 2024-09-21T23:50:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 44 2024-09-21T23:50:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:50:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:50:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:50:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:50:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:50:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:50:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:50:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:50:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:50:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:50:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:50:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:50:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:51:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 46 2024-09-21T23:51:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:00.776 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:51:00.771+0000 7f54f2d95640 -1 reset not still connected to 0x5617f33931e0 2024-09-21T23:51:00.776 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:51:00.771+0000 7f54f2d95640 -1 reset not still connected to 0x5617f498d450 2024-09-21T23:51:00.776 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:51:00.771+0000 7f54f2d95640 -1 reset not still connected to 0x5617f59a8ea0 2024-09-21T23:51:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 43 2024-09-21T23:51:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:51:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:51:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:51:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:51:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:51:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:51:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 47 2024-09-21T23:51:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 45 2024-09-21T23:51:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:51:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:51:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:51:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:51:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:51:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:51:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 48 2024-09-21T23:51:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 49 2024-09-21T23:51:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:51:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:51:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:51:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:51:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:51:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:51:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 51 2024-09-21T23:51:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 50 2024-09-21T23:51:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:51:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:51:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:51:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:51:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:51:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:51:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 52 2024-09-21T23:51:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 53 2024-09-21T23:51:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:51:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:51:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:51:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:51:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:51:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:51:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 54 2024-09-21T23:51:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 55 2024-09-21T23:51:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:51:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:51:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:51:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:51:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:51:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:51:06.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 56 2024-09-21T23:51:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:51:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:51:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:51:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 57 2024-09-21T23:51:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:51:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:51:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:51:06.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 58 2024-09-21T23:51:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 59 2024-09-21T23:51:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:51:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:51:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:51:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:51:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:51:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:51:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 60 2024-09-21T23:51:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 61 2024-09-21T23:51:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:51:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:51:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:51:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:51:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:51:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:51:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 62 2024-09-21T23:51:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 63 2024-09-21T23:51:09.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:09.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:51:09.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:09.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:09.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:51:09.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:51:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:51:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:51:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:51:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 65 2024-09-21T23:51:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 64 2024-09-21T23:51:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:51:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:51:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:51:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:51:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:51:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:51:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:10.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:51:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:51:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:51:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:51:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:51:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:51:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 67 2024-09-21T23:51:11.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 66 2024-09-21T23:51:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 68 2024-09-21T23:51:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 69 2024-09-21T23:51:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:51:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:51:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:51:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:51:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:51:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:51:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 70 2024-09-21T23:51:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 71 2024-09-21T23:51:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:51:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:51:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:51:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:51:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:51:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:51:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 72 2024-09-21T23:51:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 73 2024-09-21T23:51:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:51:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:51:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:51:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:51:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:51:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:51:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 75 2024-09-21T23:51:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 74 2024-09-21T23:51:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:51:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:51:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:51:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:51:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:51:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:51:15.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:15.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:51:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:51:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:51:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:51:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:51:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:51:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 76 2024-09-21T23:51:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 77 2024-09-21T23:51:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 78 2024-09-21T23:51:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:51:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:51:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:51:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 79 2024-09-21T23:51:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:51:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:51:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:51:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 81 2024-09-21T23:51:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 80 2024-09-21T23:51:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:51:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:51:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:51:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:51:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:51:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:51:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 83 2024-09-21T23:51:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:51:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:51:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:51:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:51:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:51:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:51:20.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 84 2024-09-21T23:51:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 85 2024-09-21T23:51:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:51:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:51:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:51:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:51:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:51:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:51:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 86 2024-09-21T23:51:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 82 2024-09-21T23:51:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:51:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:51:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:51:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:51:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:51:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:51:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 88 2024-09-21T23:51:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 87 2024-09-21T23:51:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:51:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:51:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:51:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 91 2024-09-21T23:51:23.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 90 2024-09-21T23:51:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 89 2024-09-21T23:51:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:51:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:51:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:51:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:51:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:51:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:51:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 92 2024-09-21T23:51:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:51:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:51:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:51:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:51:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:51:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:51:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 93 2024-09-21T23:51:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 94 2024-09-21T23:51:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:51:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:51:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:51:25.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:25.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:51:25.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:25.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:25.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:51:25.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:51:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:51:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:51:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:51:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 95 2024-09-21T23:51:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 96 2024-09-21T23:51:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:51:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:51:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:51:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:51:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:51:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:51:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 97 2024-09-21T23:51:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:51:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:51:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:51:27.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 98 2024-09-21T23:51:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 101 2024-09-21T23:51:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 99 2024-09-21T23:51:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 100 2024-09-21T23:51:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:51:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:51:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:51:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:51:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:51:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:51:29.617 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-21T23:51:29.607+0000 7fab1f496640 -1 reset not still connected to 0x55fa5fe39ba0 2024-09-21T23:51:29.619 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-21T23:51:29.607+0000 7fab1f496640 -1 reset not still connected to 0x55fa608ee680 2024-09-21T23:51:29.620 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-21T23:51:29.607+0000 7fab1f496640 -1 reset not still connected to 0x55fa608ef450 2024-09-21T23:51:29.620 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-21T23:51:29.607+0000 7fab1f496640 -1 reset not still connected to 0x55fa60a0e0d0 2024-09-21T23:51:29.620 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-21T23:51:29.607+0000 7fab1f496640 -1 reset not still connected to 0x55fa60a0e9c0 2024-09-21T23:51:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 102 2024-09-21T23:51:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:51:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:51:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:51:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:51:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:51:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:51:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 104 2024-09-21T23:51:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 103 2024-09-21T23:51:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:51:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:51:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:51:30.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:30.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:51:30.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:30.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:30.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:51:30.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:51:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 106 2024-09-21T23:51:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:51:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:51:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:51:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:51:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:51:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:51:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:51:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:51:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:51:31.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 105 2024-09-21T23:51:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 108 2024-09-21T23:51:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:51:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:51:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:51:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 107 2024-09-21T23:51:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 110 2024-09-21T23:51:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 111 2024-09-21T23:51:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 109 2024-09-21T23:51:33.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:51:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:51:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:51:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:51:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:51:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:51:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 112 2024-09-21T23:51:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:34.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:34.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:51:34.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:34.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:34.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:51:34.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:51:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:51:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:51:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:51:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 113 2024-09-21T23:51:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 114 2024-09-21T23:51:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:51:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:51:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:51:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:51:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:51:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:51:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:51:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:51:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:51:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:51:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:51:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:51:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:51:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:51:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:51:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 115 2024-09-21T23:51:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 116 2024-09-21T23:51:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 117 2024-09-21T23:51:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:51:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:51:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:51:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 118 2024-09-21T23:51:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 120 2024-09-21T23:51:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 119 2024-09-21T23:51:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 121 2024-09-21T23:51:38.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:38.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:38.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:51:38.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:38.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:38.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:51:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:51:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:51:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:51:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:51:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 122 2024-09-21T23:51:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:51:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:51:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:51:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:51:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:51:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:51:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 123 2024-09-21T23:51:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 124 2024-09-21T23:51:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:51:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:51:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:51:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:51:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:51:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:51:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:51:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:51:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:51:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:51:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:51:41.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:51:41.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:41.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:51:41.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:41.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:41.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:51:41.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:51:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 125 2024-09-21T23:51:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 126 2024-09-21T23:51:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:42.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 128 2024-09-21T23:51:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:51:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:51:42.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:51:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 127 2024-09-21T23:51:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 131 2024-09-21T23:51:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 129 2024-09-21T23:51:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 130 2024-09-21T23:51:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:51:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:51:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:51:43.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:51:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:51:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:51:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:51:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:51:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:51:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:51:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:51:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:51:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 132 2024-09-21T23:51:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 133 2024-09-21T23:51:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:51:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:51:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:51:45.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 134 2024-09-21T23:51:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:51:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:51:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:51:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:51:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:51:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:51:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:51:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:51:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:51:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:51:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:51:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:51:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 136 2024-09-21T23:51:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 135 2024-09-21T23:51:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 138 2024-09-21T23:51:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 137 2024-09-21T23:51:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:51:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:51:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:51:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 141 2024-09-21T23:51:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 140 2024-09-21T23:51:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 139 2024-09-21T23:51:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:51:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:51:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:51:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:51:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:51:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:51:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 142 2024-09-21T23:51:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:51:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:51:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:51:49.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:49.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:51:49.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:49.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:49.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:51:49.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:51:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 143 2024-09-21T23:51:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 144 2024-09-21T23:51:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:51:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:51:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:51:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:51:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:51:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:51:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 145 2024-09-21T23:51:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 146 2024-09-21T23:51:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:51:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:51:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:51:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:51:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:51:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:51:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:51:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:51:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:51:51.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 147 2024-09-21T23:51:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 148 2024-09-21T23:51:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:51:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:51:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:51:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 149 2024-09-21T23:51:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 150 2024-09-21T23:51:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 151 2024-09-21T23:51:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:51:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:51:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:51:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:51:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:51:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:51:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:54.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 152 2024-09-21T23:51:54.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:54.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:51:54.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:54.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:51:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:51:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:51:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:51:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:51:54.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 153 2024-09-21T23:51:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 154 2024-09-21T23:51:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:51:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:51:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:51:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:51:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:51:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:51:55.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:51:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:51:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:51:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:51:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:51:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:51:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 156 2024-09-21T23:51:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:51:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:51:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:51:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 155 2024-09-21T23:51:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 157 2024-09-21T23:51:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 158 2024-09-21T23:51:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:51:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:51:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:51:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:58.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 160 2024-09-21T23:51:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 161 2024-09-21T23:51:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 159 2024-09-21T23:51:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:51:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:51:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:51:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:51:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:51:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:51:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 162 2024-09-21T23:51:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:51:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:51:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:51:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:51:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:51:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:51:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:51:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:51:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:51:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:52:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 163 2024-09-21T23:52:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 164 2024-09-21T23:52:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:52:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:52:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:52:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:52:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:52:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:52:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 166 2024-09-21T23:52:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:52:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:52:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:52:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:52:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:52:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:52:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:52:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:52:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:52:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 165 2024-09-21T23:52:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:01.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 167 2024-09-21T23:52:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 168 2024-09-21T23:52:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:52:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:52:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:52:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 171 2024-09-21T23:52:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 170 2024-09-21T23:52:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 169 2024-09-21T23:52:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:52:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:52:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:52:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:52:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:52:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:52:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 172 2024-09-21T23:52:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:52:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:52:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:52:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:52:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:52:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:52:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 173 2024-09-21T23:52:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:52:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:52:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:52:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 174 2024-09-21T23:52:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:52:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:52:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:52:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:52:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:52:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:52:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:52:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:52:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:52:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:52:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:52:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:52:06.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 175 2024-09-21T23:52:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 176 2024-09-21T23:52:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 177 2024-09-21T23:52:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:52:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:52:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:52:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 181 2024-09-21T23:52:08.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 178 2024-09-21T23:52:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 179 2024-09-21T23:52:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 180 2024-09-21T23:52:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:52:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:52:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:52:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:52:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:52:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:52:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 182 2024-09-21T23:52:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:52:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:52:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:52:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:52:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:52:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:52:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 184 2024-09-21T23:52:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:52:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:52:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:52:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:11.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:52:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:52:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:52:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:52:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:52:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:52:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:52:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:52:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:52:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:52:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:52:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:52:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 183 2024-09-21T23:52:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 186 2024-09-21T23:52:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 187 2024-09-21T23:52:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 185 2024-09-21T23:52:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:52:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:52:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:52:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 190 2024-09-21T23:52:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 191 2024-09-21T23:52:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 189 2024-09-21T23:52:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 188 2024-09-21T23:52:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:13.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:13.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:52:13.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:13.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:13.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:52:13.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:52:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 192 2024-09-21T23:52:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:52:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:52:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:52:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:52:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:52:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:52:15.382 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-21T23:52:15.379+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a36314a9c0 2024-09-21T23:52:15.382 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-21T23:52:15.379+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a3641924e0 2024-09-21T23:52:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 193 2024-09-21T23:52:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 194 2024-09-21T23:52:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:15.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:15.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:52:15.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:15.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:15.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:52:15.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:52:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:52:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:52:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:52:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:52:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:52:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:52:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 195 2024-09-21T23:52:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 197 2024-09-21T23:52:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:52:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:52:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:52:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:52:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:52:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:52:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:52:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:52:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:52:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 196 2024-09-21T23:52:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 198 2024-09-21T23:52:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:52:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:52:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:52:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 199 2024-09-21T23:52:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 200 2024-09-21T23:52:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 201 2024-09-21T23:52:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:52:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:52:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:52:18.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:52:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:52:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:52:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 202 2024-09-21T23:52:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:52:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:52:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:52:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:52:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:52:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:52:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:52:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:52:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:52:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 203 2024-09-21T23:52:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 204 2024-09-21T23:52:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:52:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:52:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:52:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 205 2024-09-21T23:52:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:52:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:52:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:52:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:52:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:52:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:52:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 206 2024-09-21T23:52:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:52:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:52:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:52:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 208 2024-09-21T23:52:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:52:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:52:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:52:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 207 2024-09-21T23:52:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:23.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 211 2024-09-21T23:52:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 209 2024-09-21T23:52:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 210 2024-09-21T23:52:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:52:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:52:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:52:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:52:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:52:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:52:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 212 2024-09-21T23:52:24.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:24.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:52:24.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:24.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:24.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:52:24.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:52:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:52:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:52:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:52:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 214 2024-09-21T23:52:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 213 2024-09-21T23:52:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:52:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:52:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:52:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:52:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:52:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:52:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:52:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:52:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:52:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:52:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:52:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:52:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:52:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:52:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:52:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 216 2024-09-21T23:52:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 215 2024-09-21T23:52:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 218 2024-09-21T23:52:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:52:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:52:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:52:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 221 2024-09-21T23:52:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 219 2024-09-21T23:52:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 217 2024-09-21T23:52:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 220 2024-09-21T23:52:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:52:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:52:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:52:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:52:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:52:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:52:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 222 2024-09-21T23:52:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:52:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:52:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:52:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:52:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:52:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:52:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 224 2024-09-21T23:52:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 223 2024-09-21T23:52:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:52:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:52:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:52:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:52:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:52:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:52:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:52:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:52:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:52:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 226 2024-09-21T23:52:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 225 2024-09-21T23:52:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 227 2024-09-21T23:52:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:52:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:52:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:52:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 229 2024-09-21T23:52:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 228 2024-09-21T23:52:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 230 2024-09-21T23:52:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 231 2024-09-21T23:52:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:52:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:52:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:52:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:52:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:52:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:52:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 232 2024-09-21T23:52:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:52:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:52:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:52:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:52:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:52:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:52:34.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 233 2024-09-21T23:52:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 234 2024-09-21T23:52:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:52:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:52:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:52:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:52:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:52:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:52:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:52:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:52:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:52:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:52:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:52:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:52:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:52:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:52:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:52:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 235 2024-09-21T23:52:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 236 2024-09-21T23:52:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 237 2024-09-21T23:52:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 238 2024-09-21T23:52:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:52:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:52:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:52:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:38.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 241 2024-09-21T23:52:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 240 2024-09-21T23:52:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 239 2024-09-21T23:52:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:52:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:52:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:52:38.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:38.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:52:38.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:38.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:38.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:52:38.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:52:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 242 2024-09-21T23:52:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:52:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:52:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:52:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:52:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:52:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:52:40.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 244 2024-09-21T23:52:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 243 2024-09-21T23:52:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:52:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:52:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:52:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:52:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:52:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:52:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:41.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:41.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:52:41.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:41.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:41.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:52:41.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:52:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:52:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:52:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:52:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 245 2024-09-21T23:52:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 246 2024-09-21T23:52:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 247 2024-09-21T23:52:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 248 2024-09-21T23:52:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:52:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:52:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:52:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:42.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 249 2024-09-21T23:52:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 250 2024-09-21T23:52:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 251 2024-09-21T23:52:43.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:52:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:52:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:52:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:52:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:52:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:52:44.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 252 2024-09-21T23:52:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:52:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:52:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:52:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:52:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:52:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:52:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 253 2024-09-21T23:52:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 254 2024-09-21T23:52:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:52:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:52:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:52:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:52:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:52:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:52:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 255 2024-09-21T23:52:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:52:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:52:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:52:46.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 256 2024-09-21T23:52:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:52:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:52:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:52:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:52:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:52:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:52:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:52:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:52:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:52:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 257 2024-09-21T23:52:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 258 2024-09-21T23:52:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 260 2024-09-21T23:52:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 259 2024-09-21T23:52:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 261 2024-09-21T23:52:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:52:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:52:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:52:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:52:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:52:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:52:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 262 2024-09-21T23:52:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:52:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:52:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:52:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:52:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:52:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:52:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 263 2024-09-21T23:52:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 264 2024-09-21T23:52:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:52:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:52:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:52:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:52:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:52:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:52:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 265 2024-09-21T23:52:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 266 2024-09-21T23:52:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:52:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:52:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:52:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:52:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:52:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:52:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:52:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:52:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:52:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 267 2024-09-21T23:52:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:52:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:52:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:52:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 268 2024-09-21T23:52:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 269 2024-09-21T23:52:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 271 2024-09-21T23:52:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 270 2024-09-21T23:52:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:52:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:52:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:52:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:52:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:52:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:52:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 272 2024-09-21T23:52:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:52:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:52:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:52:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:52:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:52:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:52:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 273 2024-09-21T23:52:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 274 2024-09-21T23:52:55.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:52:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:52:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:52:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:52:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:52:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:52:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:52:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:52:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:52:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:52:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:52:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:52:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:52:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:52:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:52:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 276 2024-09-21T23:52:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 275 2024-09-21T23:52:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 277 2024-09-21T23:52:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 278 2024-09-21T23:52:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:52:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:52:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:52:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:58.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 279 2024-09-21T23:52:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 280 2024-09-21T23:52:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 281 2024-09-21T23:52:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:52:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:52:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:52:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:52:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:52:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:52:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 282 2024-09-21T23:52:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:52:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:52:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:52:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:52:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:52:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:52:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:52:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:52:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:52:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:52:59.847 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-21T23:52:59.842+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7c7bae680 2024-09-21T23:52:59.847 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-21T23:52:59.842+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7cd116680 2024-09-21T23:53:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 284 2024-09-21T23:53:00.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:53:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:53:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:53:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 283 2024-09-21T23:53:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:53:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:00.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:53:00.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:53:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:00.983 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-21T23:53:00.978+0000 7fab5fe19640 -1 reset not still connected to 0x5623fa61a750 2024-09-21T23:53:01.738 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-21T23:53:01.731+0000 7f84953c5640 -1 reset not still connected to 0x5636f8ec80d0 2024-09-21T23:53:01.741 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-21T23:53:01.731+0000 7f84953c5640 -1 reset not still connected to 0x5636f9c289c0 2024-09-21T23:53:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:53:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:53:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:53:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:53:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:53:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:53:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:53:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:53:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:53:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 285 2024-09-21T23:53:01.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 286 2024-09-21T23:53:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:01.846 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-21T23:53:01.842+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7c8c26340 2024-09-21T23:53:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:53:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:53:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:53:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 288 2024-09-21T23:53:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 287 2024-09-21T23:53:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:02.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 289 2024-09-21T23:53:03.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 290 2024-09-21T23:53:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 291 2024-09-21T23:53:03.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:53:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:53:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:53:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:53:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:53:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:53:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 292 2024-09-21T23:53:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:53:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:53:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:53:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:53:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:53:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:53:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 293 2024-09-21T23:53:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 294 2024-09-21T23:53:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:53:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:53:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:53:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:53:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:53:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:53:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:53:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:53:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:53:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:53:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:53:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:53:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:53:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:53:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:53:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 296 2024-09-21T23:53:06.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 295 2024-09-21T23:53:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:06.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:53:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:53:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:53:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 298 2024-09-21T23:53:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 297 2024-09-21T23:53:07.867 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-21T23:53:07.862+0000 7fab5fe19640 -1 reset not still connected to 0x5623faa15a00 2024-09-21T23:53:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 300 2024-09-21T23:53:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 299 2024-09-21T23:53:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 301 2024-09-21T23:53:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:08.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:53:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:53:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:53:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:53:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:53:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:53:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 302 2024-09-21T23:53:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:53:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:53:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:53:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:53:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:53:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:53:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 304 2024-09-21T23:53:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 303 2024-09-21T23:53:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:53:10.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:10.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:10.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:53:10.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:53:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-21T23:53:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 30727 2024-09-21T23:53:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:53:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:53:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:53:11.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-21T23:53:11.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 30862 2024-09-21T23:53:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:53:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:53:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:53:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 305 2024-09-21T23:53:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 306 2024-09-21T23:53:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 307 2024-09-21T23:53:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-21T23:53:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 30989 2024-09-21T23:53:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:13.700 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:53:13.694+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3280f70 2024-09-21T23:53:13.701 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:53:13.694+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3392ea0 2024-09-21T23:53:13.703 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:53:13.694+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3393110 2024-09-21T23:53:13.703 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:53:13.694+0000 7f54f2d95640 -1 reset not still connected to 0x5617f33931e0 2024-09-21T23:53:13.703 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:53:13.694+0000 7f54f2d95640 -1 reset not still connected to 0x5617f498d450 2024-09-21T23:53:13.703 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:53:13.694+0000 7f54f2d95640 -1 reset not still connected to 0x5617f59a8ea0 2024-09-21T23:53:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 308 2024-09-21T23:53:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 309 2024-09-21T23:53:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:53:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:53:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:53:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-21T23:53:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 31136 2024-09-21T23:53:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:53:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:53:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:53:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:53:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:53:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:53:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:53:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:53:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:53:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 310 2024-09-21T23:53:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-21T23:53:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 31283 2024-09-21T23:53:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-21T23:53:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 31431 2024-09-21T23:53:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-21T23:53:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 31578 2024-09-21T23:53:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 312 2024-09-21T23:53:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 311 2024-09-21T23:53:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:53:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-21T23:53:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 31727 2024-09-21T23:53:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-21T23:53:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 31895 2024-09-21T23:53:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-21T23:53:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-21T23:53:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 30601 2024-09-21T23:55:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-21T23:55:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-21T23:55:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-21T23:55:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 30728 2024-09-21T23:55:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-21T23:55:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-21T23:55:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-21T23:55:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 30863 2024-09-21T23:55:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-21T23:55:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-21T23:55:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-21T23:55:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 30990 2024-09-21T23:55:45.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-21T23:55:45.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-21T23:55:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-21T23:55:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 31137 2024-09-21T23:55:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-21T23:55:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-21T23:55:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-21T23:55:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 31284 2024-09-21T23:55:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-21T23:55:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-21T23:55:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-21T23:55:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 31432 2024-09-21T23:55:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-21T23:55:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-21T23:55:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-21T23:55:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 31579 2024-09-21T23:55:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-21T23:55:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-21T23:55:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-21T23:55:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 31728 2024-09-21T23:55:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-21T23:55:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-21T23:55:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-21T23:55:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 31896 2024-09-21T23:55:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-21T23:55:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-21T23:55:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-21T23:55:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:55:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-21T23:55:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:55:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 1 2024-09-21T23:55:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-21T23:55:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:55:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd0 2024-09-21T23:55:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary1 2024-09-21T23:55:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary1 2024-09-21T23:55:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 2 2024-09-21T23:55:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-21T23:55:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary1 2024-09-21T23:55:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd1 2024-09-21T23:55:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary2 2024-09-21T23:55:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary2 2024-09-21T23:55:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-21T23:55:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary2 2024-09-21T23:55:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-21T23:55:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:55:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 3 2024-09-21T23:55:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-21T23:55:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:55:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd2 2024-09-21T23:55:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 4 2024-09-21T23:55:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary3 2024-09-21T23:55:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary3 2024-09-21T23:55:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-21T23:55:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary3 2024-09-21T23:55:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd3 2024-09-21T23:55:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary4 2024-09-21T23:55:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-21T23:55:51.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary4 2024-09-21T23:55:51.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-21T23:55:51.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary4 2024-09-21T23:55:51.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:55:51.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 5 2024-09-21T23:55:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-21T23:55:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd4 2024-09-21T23:55:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary5 2024-09-21T23:55:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:55:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary5 2024-09-21T23:55:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-21T23:55:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary5 2024-09-21T23:55:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 6 2024-09-21T23:55:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd5 2024-09-21T23:55:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-21T23:55:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:55:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 7 2024-09-21T23:55:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary6 2024-09-21T23:55:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary6 2024-09-21T23:55:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd6 2024-09-21T23:55:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary7 2024-09-21T23:55:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary7 2024-09-21T23:55:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-21T23:55:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-21T23:55:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary7 2024-09-21T23:55:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:55:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-21T23:55:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary6 2024-09-21T23:55:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-21T23:55:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:55:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-21T23:55:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-21T23:55:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 37904 2024-09-21T23:55:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 10 2024-09-21T23:55:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd9 2024-09-21T23:55:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary10 2024-09-21T23:55:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary10 2024-09-21T23:55:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-21T23:55:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary10 2024-09-21T23:55:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 9 2024-09-21T23:55:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd8 2024-09-21T23:55:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary9 2024-09-21T23:55:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 8 2024-09-21T23:55:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary9 2024-09-21T23:55:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-21T23:55:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd7 2024-09-21T23:55:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary8 2024-09-21T23:55:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary8 2024-09-21T23:55:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-21T23:55:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary8 2024-09-21T23:55:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary9 2024-09-21T23:55:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-21T23:55:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:55:51.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-21T23:55:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:55:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-21T23:55:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:55:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-21T23:55:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:55:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-21T23:55:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:55:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-21T23:55:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:55:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-21T23:55:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-21T23:55:51.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:55:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-21T23:55:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-21T23:55:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:55:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:55:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:56:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=b0408cbf429b7855b3c85eff1edceaee 2024-09-21T23:56:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=551aad30d8d61994961f6984555e66c4 2024-09-21T23:56:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-21T23:56:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-21T23:56:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=e3e0fc9ff26e9e9a25c81141d9572e55 2024-09-21T23:56:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-21T23:56:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=984c8852ea068402c589b336bfce2c52 2024-09-21T23:56:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-21T23:56:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=a678d42fd1ad21b69a1f7b304bdd0d3b 2024-09-21T23:56:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-21T23:56:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=65d9b1fbccd911b91d5c34dc97fdedcb 2024-09-21T23:56:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-21T23:56:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=58ffbab66088a1426daa2870fe4204f1 2024-09-21T23:56:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-21T23:56:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=18b2e558f2d47db994710c328391e8b2 2024-09-21T23:56:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-21T23:56:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-21T23:56:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:56:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-21T23:56:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-21T23:56:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-21T23:56:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:56:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:56:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-21T23:56:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-21T23:56:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-21T23:56:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-21T23:56:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:56:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:56:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-21T23:56:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-21T23:56:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-21T23:56:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:56:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-21T23:56:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-21T23:56:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:56:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-21T23:56:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-21T23:56:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:56:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-21T23:56:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=f3b41021213a1a0b4c894baa0021202f 2024-09-21T23:56:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-21T23:56:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=43fc14eaffeae1d9026e9f5ee8796df6 2024-09-21T23:56:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-21T23:56:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-21T23:56:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:56:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-21T23:56:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-21T23:56:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:56:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-21T23:56:26.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-21T23:56:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-21T23:56:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-21T23:56:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-21T23:56:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-21T23:56:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-21T23:56:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-21T23:56:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-21T23:56:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:56:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-21T23:56:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-21T23:56:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:56:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-21T23:56:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:56:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-21T23:56:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-21T23:56:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:56:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-21T23:56:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:56:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-21T23:56:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:56:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-21T23:56:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:56:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-21T23:56:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-21T23:56:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:56:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-21T23:56:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:56:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-21T23:56:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:56:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-21T23:56:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-21T23:56:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-21T23:56:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:56:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:56:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:56:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-21T23:56:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-21T23:56:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-21T23:56:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-21T23:56:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-21T23:56:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:56:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:56:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:28.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:28.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:28.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:28.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:28.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:28.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:28.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-21T23:56:28.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:28.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-21T23:56:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-21T23:56:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:56:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-21T23:56:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:56:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-21T23:56:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-21T23:56:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-21T23:56:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 0c8e1f25-bb70-411d-ae70-cb8bd563725c 2024-09-21T23:56:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-21T23:56:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:27' 2024-09-21T23:56:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-21T23:56:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:56:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-21T23:56:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-21T23:56:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f4db6039-7ed4-4bde-a08f-e5c3a52196dd 2024-09-21T23:56:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-21T23:56:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3606528.0,"last_snapshot_bytes":7049216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962791,"remote_snapshot_timestamp":1726962791,"replay_state":"idle"} 2024-09-21T23:56:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:05' 2024-09-21T23:56:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-21T23:56:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 840fe61d-df40-46a1-b554-3a9c492652e1 2024-09-21T23:56:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-21T23:56:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2766848.0,"last_snapshot_bytes":118784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962789,"remote_snapshot_timestamp":1726962789,"replay_state":"idle"} 2024-09-21T23:56:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:05' 2024-09-21T23:56:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-21T23:56:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:56:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-21T23:56:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-21T23:56:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-21T23:56:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-21T23:56:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-21T23:56:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:56:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-21T23:56:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-21T23:56:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: caa38a51-cb46-43c6-b6d7-895eff46ab74 2024-09-21T23:56:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-21T23:56:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1730560.0,"last_snapshot_bytes":1638400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962794,"remote_snapshot_timestamp":1726962794,"replay_state":"idle"} 2024-09-21T23:56:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:05' 2024-09-21T23:56:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-21T23:56:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-21T23:56:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6ae0ec6a-c621-4b56-b1bb-8c919d2e9972 2024-09-21T23:56:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-21T23:56:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:27' 2024-09-21T23:56:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-21T23:56:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: afd18b9a-3089-4aac-8192-22348f467f8e 2024-09-21T23:56:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-21T23:56:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2117632.0,"last_snapshot_bytes":2547712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962792,"remote_snapshot_timestamp":1726962792,"replay_state":"idle"} 2024-09-21T23:56:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:05' 2024-09-21T23:56:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-21T23:56:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-21T23:56:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-21T23:56:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6697eeff-3345-479b-b250-0bb39fb9675c 2024-09-21T23:56:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-21T23:56:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:27' 2024-09-21T23:56:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-21T23:56:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-21T23:56:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 839cdadb-f02d-4927-8770-46a1da127cb6 2024-09-21T23:56:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-21T23:56:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1687552.0,"last_snapshot_bytes":1556480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962788,"remote_snapshot_timestamp":1726962788,"replay_state":"idle"} 2024-09-21T23:56:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:05' 2024-09-21T23:56:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-21T23:56:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 13a89cd3-4094-4d4b-aa75-2fadc75b5a12 2024-09-21T23:56:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-21T23:56:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1683456.0,"last_snapshot_bytes":1556480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962787,"remote_snapshot_timestamp":1726962787,"replay_state":"idle"} 2024-09-21T23:56:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:05' 2024-09-21T23:56:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-21T23:56:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-21T23:56:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-21T23:56:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 5a998568-0063-4af3-8f52-846373db4249 2024-09-21T23:56:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-21T23:56:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":931840.0,"last_snapshot_bytes":1675264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962786,"remote_snapshot_timestamp":1726962786,"replay_state":"idle"} 2024-09-21T23:56:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:05' 2024-09-21T23:56:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-21T23:56:28.561 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-21T23:56:28.541+0000 7f84953c5640 -1 reset not still connected to 0x5636f9c289c0 2024-09-21T23:56:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-21T23:56:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:56:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-21T23:56:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:56:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-21T23:56:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-21T23:56:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-21T23:56:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:56:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-21T23:56:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-21T23:56:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:56:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-21T23:56:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-21T23:56:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:56:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-21T23:56:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:56:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-21T23:56:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-21T23:56:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-21T23:56:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:56:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-21T23:56:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-21T23:56:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-21T23:56:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:56:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:56:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-21T23:56:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-21T23:56:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-21T23:56:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:56:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:30.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-21T23:56:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-21T23:56:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f4db6039-7ed4-4bde-a08f-e5c3a52196dd 2024-09-21T23:56:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-21T23:56:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:28' 2024-09-21T23:56:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-21T23:56:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 0c8e1f25-bb70-411d-ae70-cb8bd563725c 2024-09-21T23:56:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-21T23:56:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:27' 2024-09-21T23:56:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-21T23:56:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 840fe61d-df40-46a1-b554-3a9c492652e1 2024-09-21T23:56:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-21T23:56:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:28' 2024-09-21T23:56:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-09-21T23:56:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-09-21T23:56:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-09-21T23:56:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-21T23:56:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: caa38a51-cb46-43c6-b6d7-895eff46ab74 2024-09-21T23:56:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-21T23:56:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:28' 2024-09-21T23:56:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-09-21T23:56:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-21T23:56:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6ae0ec6a-c621-4b56-b1bb-8c919d2e9972 2024-09-21T23:56:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-21T23:56:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:27' 2024-09-21T23:56:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-09-21T23:56:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-21T23:56:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6697eeff-3345-479b-b250-0bb39fb9675c 2024-09-21T23:56:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-21T23:56:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:27' 2024-09-21T23:56:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-09-21T23:56:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-21T23:56:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: afd18b9a-3089-4aac-8192-22348f467f8e 2024-09-21T23:56:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-21T23:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:28' 2024-09-21T23:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-21T23:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 13a89cd3-4094-4d4b-aa75-2fadc75b5a12 2024-09-21T23:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-21T23:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:28' 2024-09-21T23:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-21T23:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 5a998568-0063-4af3-8f52-846373db4249 2024-09-21T23:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-21T23:56:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:28' 2024-09-21T23:56:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-09-21T23:56:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-09-21T23:56:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-09-21T23:56:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-21T23:56:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 839cdadb-f02d-4927-8770-46a1da127cb6 2024-09-21T23:56:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:56:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:56:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:25 2024-09-21T23:56:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-21T23:56:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:28' 2024-09-21T23:56:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-21T23:56:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-09-21T23:56:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-21T23:56:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:56:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-21T23:56:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-21T23:56:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:56:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-21T23:56:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-21T23:56:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:56:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-21T23:56:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-21T23:56:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:56:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-21T23:56:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-21T23:56:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:56:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-21T23:56:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-21T23:56:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:56:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-21T23:56:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-21T23:56:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:56:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-21T23:56:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:56:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-21T23:56:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-21T23:56:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-21T23:56:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:56:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-21T23:56:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-21T23:56:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 0c8e1f25-bb70-411d-ae70-cb8bd563725c 2024-09-21T23:56:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30 2024-09-21T23:56:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30' 2024-09-21T23:56:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-21T23:56:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:56:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-21T23:56:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:34.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:34.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-21 23:56:30' 2024-09-21T23:56:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:56:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:30' 2024-09-21T23:56:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-21T23:56:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 840fe61d-df40-46a1-b554-3a9c492652e1 2024-09-21T23:56:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30 2024-09-21T23:56:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30' 2024-09-21T23:56:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-21T23:56:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f4db6039-7ed4-4bde-a08f-e5c3a52196dd 2024-09-21T23:56:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30 2024-09-21T23:56:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30' 2024-09-21T23:56:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-21 23:56:30' 2024-09-21T23:56:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:56:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:30' 2024-09-21T23:56:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-21 23:56:30' 2024-09-21T23:56:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:56:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:30' 2024-09-21T23:56:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-21T23:56:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: caa38a51-cb46-43c6-b6d7-895eff46ab74 2024-09-21T23:56:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30 2024-09-21T23:56:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30' 2024-09-21T23:56:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-21 23:56:30' 2024-09-21T23:56:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:56:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:30' 2024-09-21T23:56:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-21T23:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 13a89cd3-4094-4d4b-aa75-2fadc75b5a12 2024-09-21T23:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30 2024-09-21T23:56:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30' 2024-09-21T23:56:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-21T23:56:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 5a998568-0063-4af3-8f52-846373db4249 2024-09-21T23:56:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30 2024-09-21T23:56:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30' 2024-09-21T23:56:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-21 23:56:30' 2024-09-21T23:56:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:56:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:30' 2024-09-21T23:56:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-21 23:56:30' 2024-09-21T23:56:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:56:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:30' 2024-09-21T23:56:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-21T23:56:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6697eeff-3345-479b-b250-0bb39fb9675c 2024-09-21T23:56:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30 2024-09-21T23:56:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30' 2024-09-21T23:56:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-21T23:56:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: afd18b9a-3089-4aac-8192-22348f467f8e 2024-09-21T23:56:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30 2024-09-21T23:56:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30' 2024-09-21T23:56:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-21 23:56:30' 2024-09-21T23:56:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:56:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:30' 2024-09-21T23:56:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-21 23:56:30' 2024-09-21T23:56:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:56:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:30' 2024-09-21T23:56:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-21T23:56:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 839cdadb-f02d-4927-8770-46a1da127cb6 2024-09-21T23:56:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30 2024-09-21T23:56:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30' 2024-09-21T23:56:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-21 23:56:30' 2024-09-21T23:56:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:56:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:30' 2024-09-21T23:56:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30' 2024-09-21T23:56:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-21T23:56:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30' 2024-09-21T23:56:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-21T23:56:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30' 2024-09-21T23:56:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-21T23:56:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30' 2024-09-21T23:56:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-21T23:56:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-21T23:56:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-21T23:56:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-21T23:56:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-21T23:56:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6ae0ec6a-c621-4b56-b1bb-8c919d2e9972 2024-09-21T23:56:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30 2024-09-21T23:56:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:56:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:30' 2024-09-21T23:56:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-21T23:56:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-21T23:56:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:56:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-21T23:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-21T23:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:56:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-21T23:56:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:56:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-21 23:56:30' 2024-09-21T23:56:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:56:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:56:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:30' 2024-09-21T23:56:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30' 2024-09-21T23:56:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-21T23:56:34.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-21T23:56:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-21T23:56:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:56:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30' 2024-09-21T23:56:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-21T23:56:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-21T23:56:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-21T23:56:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:56:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30' 2024-09-21T23:56:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-21T23:56:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-21T23:56:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-21T23:56:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:56:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30' 2024-09-21T23:56:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-21T23:56:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-21T23:56:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30' 2024-09-21T23:56:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-21T23:56:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-21T23:56:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:56:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-21T23:56:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-21T23:56:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:56:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42614264mirror.0smithi0602024-09-21 23:56:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-21 23:56:30' 2024-09-21T23:56:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-21T23:56:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:30' 2024-09-21T23:56:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-21T23:56:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-21T23:56:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:56:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:56:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:56:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-21T23:56:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:56:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-21T23:56:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-21T23:56:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:56:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:56:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-21T23:56:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-21T23:56:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-21T23:56:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-21T23:56:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:56:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-21T23:56:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-21T23:56:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:56:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-21T23:56:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-21T23:56:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:56:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-21T23:56:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-21T23:56:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:56:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-21T23:56:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-21T23:56:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-21T23:56:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-21T23:56:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:56:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-21T23:56:35.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-21T23:56:35.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f4db6039-7ed4-4bde-a08f-e5c3a52196dd 2024-09-21T23:56:35.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:35.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35 2024-09-21T23:56:35.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:35.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-21T23:56:35.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35' 2024-09-21T23:56:35.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:35.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-21T23:56:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 840fe61d-df40-46a1-b554-3a9c492652e1 2024-09-21T23:56:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35 2024-09-21T23:56:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-21T23:56:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35' 2024-09-21T23:56:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-21T23:56:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:56:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:56:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:56:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-21T23:56:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-21T23:56:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 0c8e1f25-bb70-411d-ae70-cb8bd563725c 2024-09-21T23:56:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35 2024-09-21T23:56:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-21T23:56:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35' 2024-09-21T23:56:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-21T23:56:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: caa38a51-cb46-43c6-b6d7-895eff46ab74 2024-09-21T23:56:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35 2024-09-21T23:56:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-21T23:56:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35' 2024-09-21T23:56:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-21 23:56:35' 2024-09-21T23:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-21 23:56:35' 2024-09-21T23:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:35' 2024-09-21T23:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:35' 2024-09-21T23:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-21T23:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 13a89cd3-4094-4d4b-aa75-2fadc75b5a12 2024-09-21T23:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35 2024-09-21T23:56:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-21T23:56:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35' 2024-09-21T23:56:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-21 23:56:35' 2024-09-21T23:56:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:56:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:35' 2024-09-21T23:56:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-21 23:56:35' 2024-09-21T23:56:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:56:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:35' 2024-09-21T23:56:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-21T23:56:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 5a998568-0063-4af3-8f52-846373db4249 2024-09-21T23:56:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35 2024-09-21T23:56:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-21T23:56:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35' 2024-09-21T23:56:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-21 23:56:35' 2024-09-21T23:56:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:56:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:35' 2024-09-21T23:56:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-21 23:56:35' 2024-09-21T23:56:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:56:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:35' 2024-09-21T23:56:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-21T23:56:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: afd18b9a-3089-4aac-8192-22348f467f8e 2024-09-21T23:56:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35 2024-09-21T23:56:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-21T23:56:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35' 2024-09-21T23:56:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-21T23:56:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 839cdadb-f02d-4927-8770-46a1da127cb6 2024-09-21T23:56:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35 2024-09-21T23:56:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-21T23:56:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35' 2024-09-21T23:56:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-21 23:56:35' 2024-09-21T23:56:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:56:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:35' 2024-09-21T23:56:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-21 23:56:35' 2024-09-21T23:56:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:56:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:35' 2024-09-21T23:56:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-21T23:56:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6ae0ec6a-c621-4b56-b1bb-8c919d2e9972 2024-09-21T23:56:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35 2024-09-21T23:56:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-21T23:56:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35' 2024-09-21T23:56:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-21T23:56:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6697eeff-3345-479b-b250-0bb39fb9675c 2024-09-21T23:56:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:56:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35 2024-09-21T23:56:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:56:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-21T23:56:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-21T23:56:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-21T23:56:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:56:35' 2024-09-21T23:56:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-21T23:56:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-21T23:56:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:35.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:56:35.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:35.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-21T23:56:35.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:56:35.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:35.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-21 23:56:35' 2024-09-21T23:56:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:56:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:35' 2024-09-21T23:56:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:56:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:56:35.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:56:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:56:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:56:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-21 23:56:35' 2024-09-21T23:56:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:56:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-21T23:56:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:56:35' 2024-09-21T23:56:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:56:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:56:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-21T23:56:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35' 2024-09-21T23:56:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-21T23:56:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-21T23:56:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35' 2024-09-21T23:56:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-21T23:56:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:56:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-21T23:56:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-21T23:56:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-21T23:56:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35' 2024-09-21T23:56:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-21T23:56:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-21T23:56:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-21T23:56:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35' 2024-09-21T23:56:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35' 2024-09-21T23:56:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-21T23:56:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-21T23:56:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-21T23:56:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:56:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-21T23:56:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-21T23:56:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-21T23:56:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:56:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-21T23:56:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-21T23:56:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-21T23:56:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-21T23:56:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-21T23:56:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:56:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-21T23:56:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-21T23:56:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-21T23:56:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:56:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-21T23:56:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-21T23:56:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35' 2024-09-21T23:56:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-21T23:56:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35' 2024-09-21T23:56:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-21T23:56:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:36.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35' 2024-09-21T23:56:36.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:36.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-21T23:56:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-21T23:56:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-21T23:56:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35' 2024-09-21T23:56:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-21T23:56:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:56:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-21T23:56:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-21T23:56:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+unknownremote image is not primary42634269mirror.0smithi0602024-09-21 23:56:35cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-21 23:56:35' 2024-09-21T23:56:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:56:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-21T23:56:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-21T23:56:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:56:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-21T23:56:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-21T23:56:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-21T23:56:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-21T23:56:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-21T23:56:36.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:36.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:36.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:56:35' 2024-09-21T23:56:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-21T23:56:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-21T23:56:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:56:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-21T23:56:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-21T23:56:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-21T23:56:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:56:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-21T23:56:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-21T23:56:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-21T23:56:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-21T23:56:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-21T23:56:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-21T23:56:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:56:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-21T23:56:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:56:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:56:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:56:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-21T23:56:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-21T23:56:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-21T23:56:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-21T23:56:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-21T23:56:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-21T23:56:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-21T23:56:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-21T23:56:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-21T23:56:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-21T23:56:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-21T23:56:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:56:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:56:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-21T23:56:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:56:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:56:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-21T23:56:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:56:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:56:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-21T23:56:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:56:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:56:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-21T23:56:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-21T23:56:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:56:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:56:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:56:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:56:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-21T23:56:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-21T23:56:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:56:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:56:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:56:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:56:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:56:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:56:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-21T23:56:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-21T23:56:36.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-21T23:56:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:56:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:56:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-21T23:56:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd2 2024-09-21T23:56:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-21T23:56:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:56:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd0 2024-09-21T23:56:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:56:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-21T23:56:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd4 2024-09-21T23:56:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-21T23:56:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd3 2024-09-21T23:56:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-21T23:56:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:56:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:56:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd1 2024-09-21T23:56:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-21T23:56:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:56:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd7 2024-09-21T23:56:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-21T23:56:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:56:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd6 2024-09-21T23:56:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-21T23:56:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:56:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd5 2024-09-21T23:56:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-21T23:56:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:56:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd9 2024-09-21T23:56:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:56:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-21T23:56:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd8 2024-09-21T23:56:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-21T23:56:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-21T23:57:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=f3b41021213a1a0b4c894baa0021202f 2024-09-21T23:57:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-21T23:57:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ f3b41021213a1a0b4c894baa0021202f != \f\3\b\4\1\0\2\1\2\1\3\a\1\a\0\b\4\c\8\9\4\b\a\a\0\0\2\1\2\0\2\f ]] 2024-09-21T23:57:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=a678d42fd1ad21b69a1f7b304bdd0d3b 2024-09-21T23:57:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-21T23:57:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ a678d42fd1ad21b69a1f7b304bdd0d3b != \a\6\7\8\d\4\2\f\d\1\a\d\2\1\b\6\9\a\1\f\7\b\3\0\4\b\d\d\0\d\3\b ]] 2024-09-21T23:57:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=43fc14eaffeae1d9026e9f5ee8796df6 2024-09-21T23:57:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-21T23:57:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 43fc14eaffeae1d9026e9f5ee8796df6 != \4\3\f\c\1\4\e\a\f\f\e\a\e\1\d\9\0\2\6\e\9\f\5\e\e\8\7\9\6\d\f\6 ]] 2024-09-21T23:57:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=551aad30d8d61994961f6984555e66c4 2024-09-21T23:57:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-21T23:57:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=18b2e558f2d47db994710c328391e8b2 2024-09-21T23:57:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-21T23:57:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 551aad30d8d61994961f6984555e66c4 != \5\5\1\a\a\d\3\0\d\8\d\6\1\9\9\4\9\6\1\f\6\9\8\4\5\5\5\e\6\6\c\4 ]] 2024-09-21T23:57:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 18b2e558f2d47db994710c328391e8b2 != \1\8\b\2\e\5\5\8\f\2\d\4\7\d\b\9\9\4\7\1\0\c\3\2\8\3\9\1\e\8\b\2 ]] 2024-09-21T23:57:10.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=984c8852ea068402c589b336bfce2c52 2024-09-21T23:57:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-21T23:57:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 984c8852ea068402c589b336bfce2c52 != \9\8\4\c\8\8\5\2\e\a\0\6\8\4\0\2\c\5\8\9\b\3\3\6\b\f\c\e\2\c\5\2 ]] 2024-09-21T23:57:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=b0408cbf429b7855b3c85eff1edceaee 2024-09-21T23:57:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-21T23:57:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ b0408cbf429b7855b3c85eff1edceaee != \b\0\4\0\8\c\b\f\4\2\9\b\7\8\5\5\b\3\c\8\5\e\f\f\1\e\d\c\e\a\e\e ]] 2024-09-21T23:57:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=e3e0fc9ff26e9e9a25c81141d9572e55 2024-09-21T23:57:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-21T23:57:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ e3e0fc9ff26e9e9a25c81141d9572e55 != \e\3\e\0\f\c\9\f\f\2\6\e\9\e\9\a\2\5\c\8\1\1\4\1\d\9\5\7\2\e\5\5 ]] 2024-09-21T23:57:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=65d9b1fbccd911b91d5c34dc97fdedcb 2024-09-21T23:57:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-21T23:57:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 65d9b1fbccd911b91d5c34dc97fdedcb != \6\5\d\9\b\1\f\b\c\c\d\9\1\1\b\9\1\d\5\c\3\4\d\c\9\7\f\d\e\d\c\b ]] 2024-09-21T23:57:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-21T23:57:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 37905 2024-09-21T23:57:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-21T23:57:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 37906 2024-09-21T23:57:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-21T23:57:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 37909 2024-09-21T23:57:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-21T23:57:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 37910 2024-09-21T23:57:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-21T23:57:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 37913 2024-09-21T23:57:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-21T23:57:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 37914 2024-09-21T23:57:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-21T23:57:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 37916 2024-09-21T23:57:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-21T23:57:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 37918 2024-09-21T23:57:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-21T23:57:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 37920 2024-09-21T23:57:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=58ffbab66088a1426daa2870fe4204f1 2024-09-21T23:57:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-21T23:57:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 58ffbab66088a1426daa2870fe4204f1 != \5\8\f\f\b\a\b\6\6\0\8\8\a\1\4\2\6\d\a\a\2\8\7\0\f\e\4\2\0\4\f\1 ]] 2024-09-21T23:57:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-21T23:57:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-21T23:57:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-21T23:57:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:57:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-21T23:57:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-21T23:57:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:57:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:57:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-21T23:57:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:57:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-21T23:57:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-21T23:57:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 5a998568-0063-4af3-8f52-846373db4249 2024-09-21T23:57:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-21T23:57:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3587686.4,"bytes_per_snapshot":71753728.0,"last_snapshot_bytes":71753728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:57:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:05 2024-09-21T23:57:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:57:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:57:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:57:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:57:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:10' 2024-09-21T23:57:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-21T23:57:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-21T23:57:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:57:12.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3587686.4,"bytes_per_snapshot":71753728.0,"last_snapshot_bytes":71753728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:12.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:57:12.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:12.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:57:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:57:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:57:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:57:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:57:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:05' 2024-09-21T23:57:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-21 23:57:05' 2024-09-21T23:57:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:57:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:57:05' 2024-09-21T23:57:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:57:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary15a998568-0063-4af3-8f52-846373db4249up+replayingreplaying, {"bytes_per_second":3587686.4,"bytes_per_snapshot":71753728.0,"last_snapshot_bytes":71753728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:10image-primary106697eeff-3345-479b-b250-0bb39fb9675cup+replayingreplaying, {"bytes_per_second":3239116.8,"bytes_per_snapshot":64782336.0,"last_snapshot_bytes":64782336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:10image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+replayingreplaying, {"bytes_per_second":3614515.2,"bytes_per_snapshot":72290304.0,"last_snapshot_bytes":72290304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:10image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+replayingreplaying, {"bytes_per_second":3612467.2,"bytes_per_snapshot":72249344.0,"last_snapshot_bytes":72249344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:10image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+replayingreplaying, {"bytes_per_second":3149414.4,"bytes_per_snapshot":62988288.0,"last_snapshot_bytes":62988288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:10image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+replayingreplaying, {"bytes_per_second":3483033.6,"bytes_per_snapshot":69660672.0,"last_snapshot_bytes":69660672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:10image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+replayingreplaying, {"bytes_per_second":3070566.4,"bytes_per_snapshot":61411328.0,"last_snapshot_bytes":61411328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:10image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+replayingreplaying, {"bytes_per_second":3236044.8,"bytes_per_snapshot":64720896.0,"last_snapshot_bytes":64720896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:10image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+replayingreplaying, {"bytes_per_second":3482214.4,"bytes_per_snapshot":69644288.0,"last_snapshot_bytes":69644288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:10image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+replayingreplaying, {"bytes_per_second":3604889.6,"bytes_per_snapshot":72097792.0,"last_snapshot_bytes":72097792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:10' 2024-09-21T23:57:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:57:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-21T23:57:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:05' 2024-09-21T23:57:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-21T23:57:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-21T23:57:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-21T23:57:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-21T23:57:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:57:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:57:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-21T23:57:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-21T23:57:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-21T23:57:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-21T23:57:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:57:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-21T23:57:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary1 2024-09-21T23:57:14.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:14.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary1 2024-09-21T23:57:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-21T23:57:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-21T23:57:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:57:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-21T23:57:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-21T23:57:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:57:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:57:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-21T23:57:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:57:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:15.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:15.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-21T23:57:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-21T23:57:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 13a89cd3-4094-4d4b-aa75-2fadc75b5a12 2024-09-21T23:57:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-21T23:57:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3614515.2,"bytes_per_snapshot":72290304.0,"last_snapshot_bytes":72290304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:57:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:05 2024-09-21T23:57:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:57:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:57:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:57:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:57:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:15' 2024-09-21T23:57:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-21T23:57:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-21T23:57:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:57:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3614515.2,"bytes_per_snapshot":72290304.0,"last_snapshot_bytes":72290304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:57:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:57:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:57:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:57:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:57:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:57:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:05' 2024-09-21T23:57:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-21 23:57:05' 2024-09-21T23:57:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:57:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:57:05' 2024-09-21T23:57:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:57:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK942614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary106697eeff-3345-479b-b250-0bb39fb9675cup+replayingreplaying, {"bytes_per_second":3239116.8,"bytes_per_snapshot":64782336.0,"last_snapshot_bytes":64782336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:15image-primary213a89cd3-4094-4d4b-aa75-2fadc75b5a12up+replayingreplaying, {"bytes_per_second":3614515.2,"bytes_per_snapshot":72290304.0,"last_snapshot_bytes":72290304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:15image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+replayingreplaying, {"bytes_per_second":3612467.2,"bytes_per_snapshot":72249344.0,"last_snapshot_bytes":72249344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:15image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+replayingreplaying, {"bytes_per_second":3149414.4,"bytes_per_snapshot":62988288.0,"last_snapshot_bytes":62988288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:15image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+replayingreplaying, {"bytes_per_second":3483033.6,"bytes_per_snapshot":69660672.0,"last_snapshot_bytes":69660672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:15image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+replayingreplaying, {"bytes_per_second":3070566.4,"bytes_per_snapshot":61411328.0,"last_snapshot_bytes":61411328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:15image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+replayingreplaying, {"bytes_per_second":3236044.8,"bytes_per_snapshot":64720896.0,"last_snapshot_bytes":64720896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:15image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+replayingreplaying, {"bytes_per_second":3482214.4,"bytes_per_snapshot":69644288.0,"last_snapshot_bytes":69644288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:15image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+replayingreplaying, {"bytes_per_second":3604889.6,"bytes_per_snapshot":72097792.0,"last_snapshot_bytes":72097792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:15' 2024-09-21T23:57:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:57:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-21T23:57:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:05' 2024-09-21T23:57:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-21T23:57:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-21T23:57:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-21T23:57:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-21T23:57:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:57:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:57:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-21T23:57:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-21T23:57:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-21T23:57:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-21T23:57:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:57:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-21T23:57:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary2 2024-09-21T23:57:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary2 2024-09-21T23:57:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-21T23:57:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-21T23:57:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:57:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-21T23:57:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-21T23:57:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:57:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:57:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-21T23:57:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:57:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-21T23:57:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-21T23:57:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 839cdadb-f02d-4927-8770-46a1da127cb6 2024-09-21T23:57:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-21T23:57:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3612467.2,"bytes_per_snapshot":72249344.0,"last_snapshot_bytes":72249344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:57:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:05 2024-09-21T23:57:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:57:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:57:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:57:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:57:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:15' 2024-09-21T23:57:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-21T23:57:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-21T23:57:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:57:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3612467.2,"bytes_per_snapshot":72249344.0,"last_snapshot_bytes":72249344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:57:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:57:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:57:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:57:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:57:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:57:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:05' 2024-09-21T23:57:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-21 23:57:05' 2024-09-21T23:57:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:57:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:57:05' 2024-09-21T23:57:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:57:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK842614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary106697eeff-3345-479b-b250-0bb39fb9675cup+replayingreplaying, {"bytes_per_second":3239116.8,"bytes_per_snapshot":64782336.0,"last_snapshot_bytes":64782336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:15image-primary3839cdadb-f02d-4927-8770-46a1da127cb6up+replayingreplaying, {"bytes_per_second":3612467.2,"bytes_per_snapshot":72249344.0,"last_snapshot_bytes":72249344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:15image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+replayingreplaying, {"bytes_per_second":3149414.4,"bytes_per_snapshot":62988288.0,"last_snapshot_bytes":62988288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:15image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+replayingreplaying, {"bytes_per_second":3483033.6,"bytes_per_snapshot":69660672.0,"last_snapshot_bytes":69660672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:15image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+replayingreplaying, {"bytes_per_second":3070566.4,"bytes_per_snapshot":61411328.0,"last_snapshot_bytes":61411328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:15image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+replayingreplaying, {"bytes_per_second":3236044.8,"bytes_per_snapshot":64720896.0,"last_snapshot_bytes":64720896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:15image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+replayingreplaying, {"bytes_per_second":3482214.4,"bytes_per_snapshot":69644288.0,"last_snapshot_bytes":69644288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:15image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+replayingreplaying, {"bytes_per_second":3604889.6,"bytes_per_snapshot":72097792.0,"last_snapshot_bytes":72097792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:15' 2024-09-21T23:57:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:57:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-21T23:57:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:05' 2024-09-21T23:57:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-21T23:57:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-21T23:57:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-21T23:57:19.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-21T23:57:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:57:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:57:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-21T23:57:19.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-21T23:57:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-21T23:57:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-21T23:57:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:57:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-21T23:57:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary3 2024-09-21T23:57:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary3 2024-09-21T23:57:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-21T23:57:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-21T23:57:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:57:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-21T23:57:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-21T23:57:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:57:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:57:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-21T23:57:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:57:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-21T23:57:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-21T23:57:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 840fe61d-df40-46a1-b554-3a9c492652e1 2024-09-21T23:57:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-21T23:57:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3149414.4,"bytes_per_snapshot":62988288.0,"last_snapshot_bytes":62988288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:57:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:05 2024-09-21T23:57:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:57:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:57:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:57:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:57:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:20' 2024-09-21T23:57:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-21T23:57:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-21T23:57:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:57:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3149414.4,"bytes_per_snapshot":62988288.0,"last_snapshot_bytes":62988288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:57:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:57:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:57:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:57:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:57:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:57:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:05' 2024-09-21T23:57:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-21 23:57:05' 2024-09-21T23:57:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:57:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:57:05' 2024-09-21T23:57:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:57:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK742614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary106697eeff-3345-479b-b250-0bb39fb9675cup+replayingreplaying, {"bytes_per_second":3239116.8,"bytes_per_snapshot":64782336.0,"last_snapshot_bytes":64782336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:20image-primary4840fe61d-df40-46a1-b554-3a9c492652e1up+replayingreplaying, {"bytes_per_second":3149414.4,"bytes_per_snapshot":62988288.0,"last_snapshot_bytes":62988288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:20image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+replayingreplaying, {"bytes_per_second":3483033.6,"bytes_per_snapshot":69660672.0,"last_snapshot_bytes":69660672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:20image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+replayingreplaying, {"bytes_per_second":3070566.4,"bytes_per_snapshot":61411328.0,"last_snapshot_bytes":61411328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:20image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+replayingreplaying, {"bytes_per_second":3236044.8,"bytes_per_snapshot":64720896.0,"last_snapshot_bytes":64720896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:20image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+replayingreplaying, {"bytes_per_second":3482214.4,"bytes_per_snapshot":69644288.0,"last_snapshot_bytes":69644288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:20image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+replayingreplaying, {"bytes_per_second":3604889.6,"bytes_per_snapshot":72097792.0,"last_snapshot_bytes":72097792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:20' 2024-09-21T23:57:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:57:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-21T23:57:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:05' 2024-09-21T23:57:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-21T23:57:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-21T23:57:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-21T23:57:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-21T23:57:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:57:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:57:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-21T23:57:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-21T23:57:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-21T23:57:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-21T23:57:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:57:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-21T23:57:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary4 2024-09-21T23:57:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary4 2024-09-21T23:57:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-21T23:57:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-21T23:57:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:57:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-21T23:57:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-21T23:57:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:57:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:57:24.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-21T23:57:24.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:24.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:24.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:57:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-21T23:57:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-21T23:57:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: afd18b9a-3089-4aac-8192-22348f467f8e 2024-09-21T23:57:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-21T23:57:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3483033.6,"bytes_per_snapshot":69660672.0,"last_snapshot_bytes":69660672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:57:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:05 2024-09-21T23:57:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:57:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:57:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:57:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:57:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:20' 2024-09-21T23:57:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-21T23:57:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-21T23:57:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:57:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3483033.6,"bytes_per_snapshot":69660672.0,"last_snapshot_bytes":69660672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:57:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:57:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:57:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:57:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:57:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:57:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:05' 2024-09-21T23:57:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-21 23:57:05' 2024-09-21T23:57:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:57:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:57:05' 2024-09-21T23:57:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:57:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK642614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary106697eeff-3345-479b-b250-0bb39fb9675cup+replayingreplaying, {"bytes_per_second":3239116.8,"bytes_per_snapshot":64782336.0,"last_snapshot_bytes":64782336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:20image-primary5afd18b9a-3089-4aac-8192-22348f467f8eup+replayingreplaying, {"bytes_per_second":3483033.6,"bytes_per_snapshot":69660672.0,"last_snapshot_bytes":69660672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:20image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+replayingreplaying, {"bytes_per_second":3070566.4,"bytes_per_snapshot":61411328.0,"last_snapshot_bytes":61411328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:20image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+replayingreplaying, {"bytes_per_second":3236044.8,"bytes_per_snapshot":64720896.0,"last_snapshot_bytes":64720896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:20image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+replayingreplaying, {"bytes_per_second":3482214.4,"bytes_per_snapshot":69644288.0,"last_snapshot_bytes":69644288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:20image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+replayingreplaying, {"bytes_per_second":3604889.6,"bytes_per_snapshot":72097792.0,"last_snapshot_bytes":72097792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:20' 2024-09-21T23:57:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:57:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-21T23:57:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:05' 2024-09-21T23:57:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-21T23:57:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-21T23:57:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-21T23:57:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-21T23:57:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:57:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:57:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-21T23:57:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-21T23:57:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-21T23:57:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-21T23:57:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:57:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-21T23:57:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary5 2024-09-21T23:57:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary5 2024-09-21T23:57:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-21T23:57:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-21T23:57:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:57:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-21T23:57:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-21T23:57:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:57:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:57:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-21T23:57:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:57:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-21T23:57:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-21T23:57:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f4db6039-7ed4-4bde-a08f-e5c3a52196dd 2024-09-21T23:57:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-21T23:57:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3070566.4,"bytes_per_snapshot":61411328.0,"last_snapshot_bytes":61411328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:57:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:05 2024-09-21T23:57:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:57:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:57:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:57:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:57:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:25' 2024-09-21T23:57:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-21T23:57:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-21T23:57:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:57:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3070566.4,"bytes_per_snapshot":61411328.0,"last_snapshot_bytes":61411328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:57:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:57:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:57:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:57:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:57:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:57:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:05' 2024-09-21T23:57:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-21 23:57:05' 2024-09-21T23:57:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:57:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:57:05' 2024-09-21T23:57:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:57:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK542614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary106697eeff-3345-479b-b250-0bb39fb9675cup+replayingreplaying, {"bytes_per_second":3239116.8,"bytes_per_snapshot":64782336.0,"last_snapshot_bytes":64782336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:25image-primary6f4db6039-7ed4-4bde-a08f-e5c3a52196ddup+replayingreplaying, {"bytes_per_second":3070566.4,"bytes_per_snapshot":61411328.0,"last_snapshot_bytes":61411328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:25image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+replayingreplaying, {"bytes_per_second":3236044.8,"bytes_per_snapshot":64720896.0,"last_snapshot_bytes":64720896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:25image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+replayingreplaying, {"bytes_per_second":3482214.4,"bytes_per_snapshot":69644288.0,"last_snapshot_bytes":69644288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:25image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+replayingreplaying, {"bytes_per_second":3604889.6,"bytes_per_snapshot":72097792.0,"last_snapshot_bytes":72097792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:25' 2024-09-21T23:57:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:57:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-21T23:57:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:05' 2024-09-21T23:57:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-21T23:57:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-21T23:57:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-21T23:57:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-21T23:57:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:57:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:57:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-21T23:57:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-21T23:57:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-21T23:57:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-21T23:57:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:57:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-21T23:57:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary6 2024-09-21T23:57:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary6 2024-09-21T23:57:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-21T23:57:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-21T23:57:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:57:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-21T23:57:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-21T23:57:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:57:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:57:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-21T23:57:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:57:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-21T23:57:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-21T23:57:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 0c8e1f25-bb70-411d-ae70-cb8bd563725c 2024-09-21T23:57:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-21T23:57:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3236044.8,"bytes_per_snapshot":64720896.0,"last_snapshot_bytes":64720896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:57:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:05 2024-09-21T23:57:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:57:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:57:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:57:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:57:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:30' 2024-09-21T23:57:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-21T23:57:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-21T23:57:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:57:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3236044.8,"bytes_per_snapshot":64720896.0,"last_snapshot_bytes":64720896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:57:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:57:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:57:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:57:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:57:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:57:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:05' 2024-09-21T23:57:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-21 23:57:05' 2024-09-21T23:57:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:57:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:57:05' 2024-09-21T23:57:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:57:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK442614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary106697eeff-3345-479b-b250-0bb39fb9675cup+replayingreplaying, {"bytes_per_second":3239116.8,"bytes_per_snapshot":64782336.0,"last_snapshot_bytes":64782336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:30image-primary70c8e1f25-bb70-411d-ae70-cb8bd563725cup+replayingreplaying, {"bytes_per_second":3236044.8,"bytes_per_snapshot":64720896.0,"last_snapshot_bytes":64720896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:30image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+replayingreplaying, {"bytes_per_second":3482214.4,"bytes_per_snapshot":69644288.0,"last_snapshot_bytes":69644288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:30image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+replayingreplaying, {"bytes_per_second":3604889.6,"bytes_per_snapshot":72097792.0,"last_snapshot_bytes":72097792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:30' 2024-09-21T23:57:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:57:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-21T23:57:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:05' 2024-09-21T23:57:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-21T23:57:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-21T23:57:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-21T23:57:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:31.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:31.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:31.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-21T23:57:31.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:57:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:57:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-21T23:57:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-21T23:57:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-21T23:57:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-21T23:57:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:57:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-21T23:57:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary7 2024-09-21T23:57:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary7 2024-09-21T23:57:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-21T23:57:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-21T23:57:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:57:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-21T23:57:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-21T23:57:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:57:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:57:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-21T23:57:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:57:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:33.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:33.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:33.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:33.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-21T23:57:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-21T23:57:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6ae0ec6a-c621-4b56-b1bb-8c919d2e9972 2024-09-21T23:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-21T23:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3482214.4,"bytes_per_snapshot":69644288.0,"last_snapshot_bytes":69644288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:05 2024-09-21T23:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:30' 2024-09-21T23:57:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-21T23:57:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-21T23:57:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:57:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3482214.4,"bytes_per_snapshot":69644288.0,"last_snapshot_bytes":69644288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:57:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:57:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:57:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:57:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:57:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:57:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:05' 2024-09-21T23:57:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-21 23:57:05' 2024-09-21T23:57:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:57:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:57:05' 2024-09-21T23:57:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:57:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary106697eeff-3345-479b-b250-0bb39fb9675cup+replayingreplaying, {"bytes_per_second":3239116.8,"bytes_per_snapshot":64782336.0,"last_snapshot_bytes":64782336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:30image-primary86ae0ec6a-c621-4b56-b1bb-8c919d2e9972up+replayingreplaying, {"bytes_per_second":3482214.4,"bytes_per_snapshot":69644288.0,"last_snapshot_bytes":69644288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:30image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+replayingreplaying, {"bytes_per_second":3604889.6,"bytes_per_snapshot":72097792.0,"last_snapshot_bytes":72097792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:30' 2024-09-21T23:57:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:57:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-21T23:57:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:05' 2024-09-21T23:57:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-21T23:57:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-21T23:57:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-21T23:57:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-21T23:57:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:57:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:57:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-21T23:57:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-21T23:57:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-21T23:57:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-21T23:57:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:57:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-21T23:57:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary8 2024-09-21T23:57:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary8 2024-09-21T23:57:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-21T23:57:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-21T23:57:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:57:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-21T23:57:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-21T23:57:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:57:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:57:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-21T23:57:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:57:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-21T23:57:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-21T23:57:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: caa38a51-cb46-43c6-b6d7-895eff46ab74 2024-09-21T23:57:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-21T23:57:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72097792.0,"last_snapshot_bytes":72097792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:57:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:35 2024-09-21T23:57:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:57:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:57:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:57:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:57:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:35' 2024-09-21T23:57:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-21T23:57:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-21T23:57:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:57:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72097792.0,"last_snapshot_bytes":72097792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:57:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:57:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:57:37.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:57:37.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:57:37.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:57:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:35' 2024-09-21T23:57:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-21 23:57:35' 2024-09-21T23:57:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:57:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:57:35' 2024-09-21T23:57:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:57:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK242614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary106697eeff-3345-479b-b250-0bb39fb9675cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64782336.0,"last_snapshot_bytes":64782336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:35image-primary9caa38a51-cb46-43c6-b6d7-895eff46ab74up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72097792.0,"last_snapshot_bytes":72097792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:35' 2024-09-21T23:57:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:57:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-21T23:57:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:35' 2024-09-21T23:57:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-21T23:57:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-21T23:57:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-21T23:57:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-21T23:57:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:57:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:57:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-21T23:57:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-21T23:57:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-21T23:57:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-21T23:57:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:57:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-21T23:57:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary9 2024-09-21T23:57:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary9 2024-09-21T23:57:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-21T23:57:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-21T23:57:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:57:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-21T23:57:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-21T23:57:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:57:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-21T23:57:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-21T23:57:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:57:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-21T23:57:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-21T23:57:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-21T23:57:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-21T23:57:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6697eeff-3345-479b-b250-0bb39fb9675c 2024-09-21T23:57:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-21T23:57:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64782336.0,"last_snapshot_bytes":64782336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-21T23:57:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:35 2024-09-21T23:57:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-21T23:57:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-21T23:57:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-21T23:57:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-21T23:57:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-21 23:57:35' 2024-09-21T23:57:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-21T23:57:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-21T23:57:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-21T23:57:40.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64782336.0,"last_snapshot_bytes":64782336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"} 2024-09-21T23:57:40.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-21T23:57:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-21T23:57:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-21T23:57:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-21T23:57:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-21T23:57:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-21T23:57:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:35' 2024-09-21T23:57:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-21 23:57:35' 2024-09-21T23:57:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:57:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-21T23:57:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-21 23:57:35' 2024-09-21T23:57:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-21T23:57:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-21T23:57:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-21T23:57:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK142614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary106697eeff-3345-479b-b250-0bb39fb9675cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64782336.0,"last_snapshot_bytes":64782336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726962996,"remote_snapshot_timestamp":1726962996,"replay_state":"idle"}42614264mirror.0smithi0602024-09-21 23:57:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-21 23:57:35' 2024-09-21T23:57:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-21T23:57:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-21T23:57:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-21 23:57:35' 2024-09-21T23:57:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-21T23:57:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-21T23:57:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-21T23:57:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-21T23:57:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-21T23:57:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-21T23:57:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-21T23:57:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:57:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-21T23:57:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-21T23:57:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-21T23:57:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-21T23:57:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:57:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-21T23:57:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-21T23:57:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary10 2024-09-21T23:57:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary10 2024-09-21T23:57:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..10} 2024-09-21T23:57:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS=() 2024-09-21T23:57:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS=() 2024-09-21T23:57:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS=() 2024-09-21T23:57:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS=() 2024-09-21T23:57:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ RET=0 2024-09-21T23:57:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-21T23:57:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary1 2024-09-21T23:57:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-21T23:57:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:57:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:57:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-21T23:57:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:57:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:57:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:57:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:57:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-21T23:57:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:57:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:57:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-21T23:57:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:57:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-21T23:57:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:57:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-21T23:57:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary1': 2024-09-21T23:57:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:57:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:57:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:57:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 154e3823cc54 2024-09-21T23:57:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.154e3823cc54 2024-09-21T23:57:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:57:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:57:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:57:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:57:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:57:42 2024 2024-09-21T23:57:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:57:42 2024 2024-09-21T23:57:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:57:42 2024 2024-09-21T23:57:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:57:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:57:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 75eb3074-7ec8-41d0-818c-5d73fe84dc99 2024-09-21T23:57:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:57:42.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:57:42.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:57:42.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-21T23:57:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd0 2024-09-21T23:57:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:57:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-21T23:57:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:57:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-21T23:57:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:57:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 0ad12f14-fe97-4f8d-98ce-5f7fc74c1459 2024-09-21T23:57:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:57:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:57:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:57:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:57:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:57:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:57:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary1 2024-09-21T23:57:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-21T23:57:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:57:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary1 2024-09-21T23:57:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:57:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-21T23:57:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:57:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:57:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:57:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary1 2024-09-21T23:57:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:57:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-21T23:57:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:57:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary2 2024-09-21T23:57:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-21T23:57:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary1 2024-09-21T23:57:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:57:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-21T23:57:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:57:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-21T23:57:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:57:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:57:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:57:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:57:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-21T23:57:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:57:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:57:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-21T23:57:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:57:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-21T23:57:43.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 334 2024-09-21T23:57:43.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:57:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-21T23:57:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary2': 2024-09-21T23:57:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:57:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:57:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:57:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 155c504735db 2024-09-21T23:57:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.155c504735db 2024-09-21T23:57:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:57:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:57:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:57:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:57:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:57:43 2024 2024-09-21T23:57:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:57:43 2024 2024-09-21T23:57:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:57:43 2024 2024-09-21T23:57:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:57:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:57:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 2d731e2c-10e1-40be-a168-8f60f68f0436 2024-09-21T23:57:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:57:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:57:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:57:43.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-21T23:57:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd1 2024-09-21T23:57:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:57:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-21T23:57:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:57:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-21T23:57:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:57:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 2c4f244b-635f-4737-971d-b4753144a18a 2024-09-21T23:57:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:57:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:57:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:57:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:57:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:57:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:57:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary2 2024-09-21T23:57:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-21T23:57:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:57:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary2 2024-09-21T23:57:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:57:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-21T23:57:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:57:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-21T23:57:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary3 2024-09-21T23:57:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary2 2024-09-21T23:57:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-21T23:57:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:57:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:57:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:57:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:57:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-21T23:57:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:57:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary2 2024-09-21T23:57:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:57:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-21T23:57:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:57:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:57:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:57:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:57:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-21T23:57:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:57:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:57:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-21T23:57:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:57:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-21T23:57:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:57:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-21T23:57:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:57:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-21T23:57:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:57:44.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 336 2024-09-21T23:57:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-21T23:57:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary3': 2024-09-21T23:57:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:57:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:57:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:57:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 156a89a8128c 2024-09-21T23:57:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.156a89a8128c 2024-09-21T23:57:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:57:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:57:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:57:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:57:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:57:44 2024 2024-09-21T23:57:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:57:44 2024 2024-09-21T23:57:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:57:44 2024 2024-09-21T23:57:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:57:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:57:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: a97fee7f-8f8d-4ae7-8828-dad7b24deda0 2024-09-21T23:57:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:57:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:57:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:57:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-21T23:57:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd2 2024-09-21T23:57:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:57:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-21T23:57:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:57:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/26214401572864/2621440 done 2024-09-21T23:57:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:57:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 21c78a58-f46d-4be0-908d-538ee153de3e 2024-09-21T23:57:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:57:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:57:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:57:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:57:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:57:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:57:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary3 2024-09-21T23:57:45.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-21T23:57:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:57:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary3 2024-09-21T23:57:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:57:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-21T23:57:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:57:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:57:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary3 2024-09-21T23:57:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:57:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:57:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-21T23:57:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary4 2024-09-21T23:57:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:57:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-21T23:57:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary3 2024-09-21T23:57:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-21T23:57:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:57:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:57:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-21T23:57:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:57:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:57:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:57:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:57:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-21T23:57:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:57:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:57:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-21T23:57:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:57:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-21T23:57:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:57:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-21T23:57:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:57:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 338 2024-09-21T23:57:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-21T23:57:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary4': 2024-09-21T23:57:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:57:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:57:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:57:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1578b9c0a7cf 2024-09-21T23:57:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1578b9c0a7cf 2024-09-21T23:57:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:57:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:57:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:57:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:57:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:57:45 2024 2024-09-21T23:57:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:57:45 2024 2024-09-21T23:57:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:57:45 2024 2024-09-21T23:57:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:57:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:57:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 74559236-7ec4-4cda-b42c-9d9866ab7ec4 2024-09-21T23:57:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:57:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:57:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:57:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-21T23:57:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd3 2024-09-21T23:57:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:57:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-21T23:57:46.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:57:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-21T23:57:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:57:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: f799c74d-ed4c-4f37-bf1c-e76a84928981 2024-09-21T23:57:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:57:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:57:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:57:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:57:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:57:46.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:57:46.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary4 2024-09-21T23:57:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-21T23:57:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:57:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary4 2024-09-21T23:57:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:57:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:57:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-21T23:57:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary5 2024-09-21T23:57:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-21T23:57:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-21T23:57:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary4 2024-09-21T23:57:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:57:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:57:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-21T23:57:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:57:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:57:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:57:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:57:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-21T23:57:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary4 2024-09-21T23:57:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:57:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-21T23:57:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:57:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:57:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:57:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:57:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:57:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-21T23:57:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:57:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-21T23:57:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:57:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-21T23:57:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:57:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:57:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:57:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:57:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 340 2024-09-21T23:57:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-21T23:57:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary5': 2024-09-21T23:57:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:57:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:57:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:57:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 15861586bedf 2024-09-21T23:57:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.15861586bedf 2024-09-21T23:57:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:57:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:57:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:57:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:57:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:57:46 2024 2024-09-21T23:57:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:57:46 2024 2024-09-21T23:57:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:57:46 2024 2024-09-21T23:57:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:57:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:57:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: e0332846-87f6-41a2-87e5-407f3baf7638 2024-09-21T23:57:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:57:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:57:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:57:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-21T23:57:47.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd4 2024-09-21T23:57:47.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:57:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-21T23:57:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:57:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-21T23:57:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:57:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 406bf0c7-7f9e-47c2-a5f2-91c906b4f4c3 2024-09-21T23:57:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:57:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:57:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:57:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:57:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:57:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:57:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary5 2024-09-21T23:57:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-21T23:57:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:57:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary5 2024-09-21T23:57:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:57:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:57:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-21T23:57:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary5 2024-09-21T23:57:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-21T23:57:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary6 2024-09-21T23:57:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-21T23:57:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:57:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:57:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:57:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary5 2024-09-21T23:57:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-21T23:57:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:57:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-21T23:57:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:57:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:57:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:57:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:57:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:57:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:57:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-21T23:57:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:57:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:57:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-21T23:57:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:57:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-21T23:57:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:57:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-21T23:57:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:57:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:57:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:57:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 343 2024-09-21T23:57:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:57:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-21T23:57:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary6': 2024-09-21T23:57:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:57:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:57:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:57:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1598caafc6ce 2024-09-21T23:57:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1598caafc6ce 2024-09-21T23:57:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:57:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:57:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:57:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:57:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:57:47 2024 2024-09-21T23:57:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:57:47 2024 2024-09-21T23:57:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:57:47 2024 2024-09-21T23:57:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:57:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:57:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 8c18f283-4807-40f7-a279-3c54d0de9982 2024-09-21T23:57:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:57:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:57:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:57:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-21T23:57:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd5 2024-09-21T23:57:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:57:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-21T23:57:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:57:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-21T23:57:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:57:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 326f0548-0e1b-465a-a6b7-8d30422dc998 2024-09-21T23:57:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:57:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:57:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:57:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:57:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:57:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:57:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary6 2024-09-21T23:57:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-21T23:57:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:57:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary6 2024-09-21T23:57:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:57:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-21T23:57:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:57:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-21T23:57:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary7 2024-09-21T23:57:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-21T23:57:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary6 2024-09-21T23:57:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:57:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:57:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-21T23:57:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:57:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:57:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:57:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary6 2024-09-21T23:57:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:57:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:57:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-21T23:57:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-21T23:57:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:57:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:57:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:57:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:57:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:57:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-21T23:57:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:57:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-21T23:57:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:57:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-21T23:57:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 342 2024-09-21T23:57:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 346 2024-09-21T23:57:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:57:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-21T23:57:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:57:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:57:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:57:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary7': 2024-09-21T23:57:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:57:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:57:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:57:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 15aa575e7a28 2024-09-21T23:57:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.15aa575e7a28 2024-09-21T23:57:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:57:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:57:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:57:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:57:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:57:48 2024 2024-09-21T23:57:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:57:48 2024 2024-09-21T23:57:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:57:48 2024 2024-09-21T23:57:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:57:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:57:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: e7e01dd6-909d-415a-85aa-040f7cd3f42c 2024-09-21T23:57:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:57:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:57:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:57:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-21T23:57:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd6 2024-09-21T23:57:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:57:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-21T23:57:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:57:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-21T23:57:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:57:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 24f36ea2-ad0b-4542-ba40-4da11956240b 2024-09-21T23:57:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:57:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:57:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:57:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:57:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:57:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:57:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary7 2024-09-21T23:57:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-21T23:57:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:57:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary7 2024-09-21T23:57:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:57:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-21T23:57:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:57:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:57:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary7 2024-09-21T23:57:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:57:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-21T23:57:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary8 2024-09-21T23:57:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-21T23:57:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:57:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:57:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:57:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:57:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary7 2024-09-21T23:57:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-21T23:57:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-21T23:57:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:57:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:57:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:57:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:57:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-21T23:57:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:57:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:57:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-21T23:57:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:57:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-21T23:57:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:57:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-21T23:57:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 345 2024-09-21T23:57:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:57:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:57:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:57:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:49.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 349 2024-09-21T23:57:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:57:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-21T23:57:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary8': 2024-09-21T23:57:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:57:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:57:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:57:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 15bc8f321a9c 2024-09-21T23:57:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.15bc8f321a9c 2024-09-21T23:57:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:57:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:57:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:57:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:57:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:57:49 2024 2024-09-21T23:57:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:57:49 2024 2024-09-21T23:57:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:57:49 2024 2024-09-21T23:57:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:57:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:57:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 2af3188f-f478-4e84-ab55-bc009d9f76ef 2024-09-21T23:57:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:57:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:57:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:57:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-21T23:57:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd7 2024-09-21T23:57:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:57:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-21T23:57:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:57:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-21T23:57:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:57:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 13874e08-d9b4-4e72-86cf-694824824a3b 2024-09-21T23:57:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:57:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:57:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:57:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:57:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:57:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:57:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary8 2024-09-21T23:57:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-21T23:57:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:57:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary8 2024-09-21T23:57:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:57:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-21T23:57:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:57:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:57:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:57:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:57:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:57:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-21T23:57:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary8 2024-09-21T23:57:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary9 2024-09-21T23:57:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-21T23:57:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:57:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary8 2024-09-21T23:57:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:57:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-21T23:57:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-21T23:57:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:57:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:50.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:57:50.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:57:50.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:57:50.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:50.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-21T23:57:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:57:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:57:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-21T23:57:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:57:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-21T23:57:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:57:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-21T23:57:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 348 2024-09-21T23:57:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:57:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:57:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:57:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:57:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 352 2024-09-21T23:57:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-21T23:57:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:51.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary9': 2024-09-21T23:57:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:57:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:57:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:57:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 15ccae0efcfd 2024-09-21T23:57:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.15ccae0efcfd 2024-09-21T23:57:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:57:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:57:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:57:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:57:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:57:50 2024 2024-09-21T23:57:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:57:50 2024 2024-09-21T23:57:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:57:50 2024 2024-09-21T23:57:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:57:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:57:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 007a3581-8ac5-40d4-afd3-4c2a0da39a63 2024-09-21T23:57:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:57:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:57:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:57:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-21T23:57:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd8 2024-09-21T23:57:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:57:51.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-21T23:57:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:57:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-21T23:57:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:57:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: b9a5636d-d18c-4f0b-9398-1a7e80595a54 2024-09-21T23:57:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:57:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:57:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:57:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:57:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:57:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:57:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary9 2024-09-21T23:57:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-21T23:57:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:57:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary9 2024-09-21T23:57:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:57:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-21T23:57:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:57:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-21T23:57:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:57:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-21T23:57:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary10 2024-09-21T23:57:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-21T23:57:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary9 2024-09-21T23:57:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:57:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:57:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:57:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:57:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-21T23:57:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-21T23:57:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary9 2024-09-21T23:57:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:57:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-21T23:57:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-21T23:57:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-21T23:57:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-21T23:57:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-21T23:57:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-21T23:57:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-21T23:57:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-21T23:57:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-21T23:57:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-21T23:57:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:57:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-21T23:57:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-21T23:57:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:57:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-21T23:57:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:57:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:57:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:57:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:57:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:57:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:57:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 351 2024-09-21T23:57:51.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-21T23:57:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 355 2024-09-21T23:57:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-21T23:57:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary10': 2024-09-21T23:57:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-21T23:57:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-21T23:57:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-21T23:57:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 15dde4de5fc6 2024-09-21T23:57:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.15dde4de5fc6 2024-09-21T23:57:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-21T23:57:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-21T23:57:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-21T23:57:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-21T23:57:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sat Sep 21 23:57:51 2024 2024-09-21T23:57:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sat Sep 21 23:57:51 2024 2024-09-21T23:57:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sat Sep 21 23:57:51 2024 2024-09-21T23:57:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-21T23:57:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-21T23:57:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 99b83aef-02ad-4e23-8d94-f5c1d21275b6 2024-09-21T23:57:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-21T23:57:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-21T23:57:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-21T23:57:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-21T23:57:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd9 2024-09-21T23:57:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-21T23:57:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-21T23:57:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T23:57:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-21T23:57:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-21T23:57:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: dc89f9bc-d950-4827-b7e0-50ed16b447d7 2024-09-21T23:57:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-21T23:57:52.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-21T23:57:52.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:52.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-21T23:57:52.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-21T23:57:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-21T23:57:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-21T23:57:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-21T23:57:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary10 2024-09-21T23:57:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-21T23:57:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-21T23:57:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary10 2024-09-21T23:57:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-21T23:57:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-21T23:57:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:57:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-21T23:57:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:57:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:57:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-21T23:57:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary10 2024-09-21T23:57:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:57:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 42075 2024-09-21T23:57:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary10 2024-09-21T23:57:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-21T23:57:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-21T23:57:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-21T23:57:52.753 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-21T23:57:52.748+0000 7f84953c5640 -1 reset not still connected to 0x5636f85f0d00 2024-09-21T23:57:52.753 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-21T23:57:52.748+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b32ea0 2024-09-21T23:57:52.753 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-21T23:57:52.748+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b332b0 2024-09-21T23:57:52.753 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-21T23:57:52.748+0000 7f84953c5640 -1 reset not still connected to 0x5636f8ec80d0 2024-09-21T23:57:52.753 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-21T23:57:52.748+0000 7f84953c5640 -1 reset not still connected to 0x5636f9c289c0 2024-09-21T23:57:52.753 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-21T23:57:52.748+0000 7f84953c5640 -1 reset not still connected to 0x5636fb5d6dd0 2024-09-21T23:57:52.753 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-21T23:57:52.748+0000 7f84953c5640 -1 reset not still connected to 0x563700648ea0 2024-09-21T23:57:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 354 2024-09-21T23:57:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:52.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 359 2024-09-21T23:57:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:57:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:57:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:57:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:57:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:57:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:57:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 358 2024-09-21T23:57:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 357 2024-09-21T23:57:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:53.802 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-21T23:57:53.784+0000 7f84953c5640 -1 reset not still connected to 0x5636f85f0a90 2024-09-21T23:57:53.802 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-21T23:57:53.784+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b332b0 2024-09-21T23:57:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:57:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:57:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:57:54.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:54.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:57:54.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:54.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:54.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:57:54.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:57:54.840 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-21T23:57:54.828+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a367be85b0 2024-09-21T23:57:54.841 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-21T23:57:54.836+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a368111790 2024-09-21T23:57:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 360 2024-09-21T23:57:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 361 2024-09-21T23:57:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:57:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:57:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:57:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:57:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:57:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:57:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 362 2024-09-21T23:57:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:57:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:57:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:57:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 363 2024-09-21T23:57:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:57:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:57:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:57:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:57:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:57:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:57:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:57:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:57:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:57:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 365 2024-09-21T23:57:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 364 2024-09-21T23:57:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 367 2024-09-21T23:57:57.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 366 2024-09-21T23:57:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:57:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:57:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:57:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:57:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:57:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:57:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 368 2024-09-21T23:57:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 369 2024-09-21T23:57:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:57:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:57:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:57:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:57:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:57:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:57:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:57:59.883 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:57:59.879+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3392ea0 2024-09-21T23:57:59.887 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:57:59.887+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3393110 2024-09-21T23:57:59.889 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:57:59.887+0000 7f54f2d95640 -1 reset not still connected to 0x5617f33931e0 2024-09-21T23:57:59.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 370 2024-09-21T23:57:59.892 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:57:59.891+0000 7f54f2d95640 -1 reset not still connected to 0x5617f4bf0340 2024-09-21T23:57:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:57:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:57:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:57:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:57:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:57:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:57:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:57:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:57:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:57:59.924 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:57:59.923+0000 7f54f2d95640 -1 reset not still connected to 0x5617f5eb6a90 2024-09-21T23:57:59.924 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:57:59.923+0000 7f54f2d95640 -1 reset not still connected to 0x5617f7c6ca90 2024-09-21T23:57:59.924 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:57:59.923+0000 7f54f2d95640 -1 reset not still connected to 0x5617fb678000 2024-09-21T23:57:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 371 2024-09-21T23:57:59.933 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:57:59.931+0000 7f54f2d95640 -1 reset not still connected to 0x5617fb742c30 2024-09-21T23:57:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:00.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:00.031 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:58:00.031+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3393110 2024-09-21T23:58:00.033 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:58:00.031+0000 7f54f2d95640 -1 reset not still connected to 0x5617f5eb6a90 2024-09-21T23:58:00.033 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:58:00.031+0000 7f54f2d95640 -1 reset not still connected to 0x5617f7c6ca90 2024-09-21T23:58:00.033 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:58:00.031+0000 7f54f2d95640 -1 reset not still connected to 0x5617fb678000 2024-09-21T23:58:00.033 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:58:00.031+0000 7f54f2d95640 -1 reset not still connected to 0x5617fb742c30 2024-09-21T23:58:00.209 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-21T23:58:00.208+0000 7fab1f496640 -1 reset not still connected to 0x55fa626dfba0 2024-09-21T23:58:00.209 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-21T23:58:00.208+0000 7fab1f496640 -1 reset not still connected to 0x55fa63cf7930 2024-09-21T23:58:00.209 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-21T23:58:00.208+0000 7fab1f496640 -1 reset not still connected to 0x55fa63f40270 2024-09-21T23:58:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 373 2024-09-21T23:58:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:58:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:58:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:58:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:58:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:58:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:58:00.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 372 2024-09-21T23:58:01.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:58:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:58:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:58:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:58:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:58:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:58:01.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 374 2024-09-21T23:58:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 375 2024-09-21T23:58:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:02.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 377 2024-09-21T23:58:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 376 2024-09-21T23:58:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:58:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:58:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:58:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:58:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:58:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 378 2024-09-21T23:58:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 379 2024-09-21T23:58:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:58:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:58:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:58:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:58:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:58:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:58:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 381 2024-09-21T23:58:04.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 380 2024-09-21T23:58:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:04.944 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-21T23:58:04.943+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7c486b2b0 2024-09-21T23:58:04.944 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-21T23:58:04.943+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7c591e410 2024-09-21T23:58:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:58:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:58:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:58:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:58:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:58:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:58:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 382 2024-09-21T23:58:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 383 2024-09-21T23:58:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:05.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:58:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:58:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:58:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:58:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:58:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:58:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 384 2024-09-21T23:58:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 385 2024-09-21T23:58:06.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:06.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:58:06.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:58:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:58:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:58:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:58:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:58:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 387 2024-09-21T23:58:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 386 2024-09-21T23:58:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:58:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:58:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:58:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:58:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:58:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:58:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 388 2024-09-21T23:58:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 389 2024-09-21T23:58:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:58:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:58:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:58:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:58:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:58:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:58:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 391 2024-09-21T23:58:09.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 390 2024-09-21T23:58:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:58:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:58:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:58:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:58:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:58:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:58:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 392 2024-09-21T23:58:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 393 2024-09-21T23:58:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:58:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:58:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:58:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:58:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:58:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:58:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 394 2024-09-21T23:58:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 395 2024-09-21T23:58:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:58:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:58:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:58:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:58:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:58:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:58:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 397 2024-09-21T23:58:12.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:12.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:58:12.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:12.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:12.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:58:12.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:58:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 396 2024-09-21T23:58:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:58:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:58:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:58:12.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 399 2024-09-21T23:58:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 398 2024-09-21T23:58:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:13.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:58:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:58:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:58:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:58:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:58:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:58:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 401 2024-09-21T23:58:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 400 2024-09-21T23:58:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:58:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:58:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:58:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:58:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:58:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:58:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:14.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:58:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:58:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:58:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:58:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:58:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:58:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 402 2024-09-21T23:58:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 403 2024-09-21T23:58:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:58:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:58:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:58:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:58:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:58:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:58:16.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 404 2024-09-21T23:58:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 405 2024-09-21T23:58:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 407 2024-09-21T23:58:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:58:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:58:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:58:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 406 2024-09-21T23:58:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:58:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:58:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:58:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 409 2024-09-21T23:58:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 408 2024-09-21T23:58:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:58:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:58:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:58:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:58:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:58:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:58:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 411 2024-09-21T23:58:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 410 2024-09-21T23:58:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:58:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:58:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:58:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:58:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:58:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:58:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 413 2024-09-21T23:58:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 412 2024-09-21T23:58:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:58:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:58:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:58:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:58:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:58:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:58:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 415 2024-09-21T23:58:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 414 2024-09-21T23:58:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:58:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:58:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:58:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:58:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:58:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:58:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 416 2024-09-21T23:58:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 417 2024-09-21T23:58:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:58:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:58:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:58:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:58:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:58:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:58:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 418 2024-09-21T23:58:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 419 2024-09-21T23:58:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:58:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:58:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:58:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:58:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:58:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:58:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 421 2024-09-21T23:58:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 420 2024-09-21T23:58:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:58:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:58:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:58:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:58:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:58:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:58:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 422 2024-09-21T23:58:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 423 2024-09-21T23:58:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:58:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:58:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:58:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:58:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:58:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:58:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 425 2024-09-21T23:58:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 424 2024-09-21T23:58:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:58:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:58:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:58:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:58:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:58:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:58:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 426 2024-09-21T23:58:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 427 2024-09-21T23:58:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:58:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:58:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:58:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:58:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:58:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:58:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 428 2024-09-21T23:58:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 429 2024-09-21T23:58:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:58:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:58:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:58:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:58:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:58:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:58:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 431 2024-09-21T23:58:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 430 2024-09-21T23:58:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:58:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:58:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:58:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:58:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:58:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:58:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 432 2024-09-21T23:58:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 433 2024-09-21T23:58:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:58:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:58:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:58:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:58:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:58:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:58:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:58:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:58:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:58:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:58:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:58:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:58:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 435 2024-09-21T23:58:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 434 2024-09-21T23:58:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 436 2024-09-21T23:58:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 437 2024-09-21T23:58:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:58:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:58:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:58:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:58:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:58:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:58:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 438 2024-09-21T23:58:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 439 2024-09-21T23:58:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:58:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:58:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:58:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:58:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:58:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:58:34.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 440 2024-09-21T23:58:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 441 2024-09-21T23:58:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:58:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:58:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:58:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:58:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:58:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:58:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:35.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 443 2024-09-21T23:58:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 442 2024-09-21T23:58:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:58:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:58:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:58:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:58:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:58:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:58:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:58:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:58:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:58:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:58:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:58:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:58:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 445 2024-09-21T23:58:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 444 2024-09-21T23:58:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:36.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 447 2024-09-21T23:58:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 446 2024-09-21T23:58:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:58:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:58:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:58:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:58:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:58:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:58:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 449 2024-09-21T23:58:38.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 448 2024-09-21T23:58:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:58:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:58:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:58:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:58:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:58:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:58:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 450 2024-09-21T23:58:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 451 2024-09-21T23:58:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:58:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:58:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:58:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:58:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:58:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:58:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 452 2024-09-21T23:58:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 453 2024-09-21T23:58:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:58:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:58:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:58:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:58:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:58:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:58:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:58:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:58:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:58:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:58:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:58:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:58:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 455 2024-09-21T23:58:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 454 2024-09-21T23:58:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 456 2024-09-21T23:58:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 457 2024-09-21T23:58:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:58:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:58:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:58:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:58:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:58:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:58:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 459 2024-09-21T23:58:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 458 2024-09-21T23:58:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:58:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:58:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:58:43.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:43.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:58:43.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:43.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:43.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:58:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:58:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 461 2024-09-21T23:58:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 460 2024-09-21T23:58:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:58:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:58:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:58:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:58:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:58:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:58:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 463 2024-09-21T23:58:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 462 2024-09-21T23:58:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:58:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:58:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:58:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:58:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:58:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:58:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 465 2024-09-21T23:58:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:58:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:58:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:58:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:58:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:58:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:58:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 467 2024-09-21T23:58:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 464 2024-09-21T23:58:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 466 2024-09-21T23:58:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:58:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:58:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:58:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:58:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:58:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:58:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 468 2024-09-21T23:58:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 469 2024-09-21T23:58:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:58:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:58:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:58:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:58:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:58:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:58:49.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 470 2024-09-21T23:58:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:58:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:58:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:58:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 471 2024-09-21T23:58:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 472 2024-09-21T23:58:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 473 2024-09-21T23:58:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:58:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:58:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:58:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:58:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:58:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:58:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:58:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:58:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:58:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 474 2024-09-21T23:58:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:58:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:58:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:58:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:58:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:58:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:58:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 476 2024-09-21T23:58:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 475 2024-09-21T23:58:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:58:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:58:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:58:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 478 2024-09-21T23:58:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 477 2024-09-21T23:58:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 479 2024-09-21T23:58:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:58:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:58:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:58:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:58:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:58:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:58:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:58:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:58:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:58:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 480 2024-09-21T23:58:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:58:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:58:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:58:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 481 2024-09-21T23:58:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:58:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:58:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:58:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:55.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 483 2024-09-21T23:58:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 482 2024-09-21T23:58:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:58:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:58:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:58:55.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 485 2024-09-21T23:58:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:58:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:58:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:58:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 484 2024-09-21T23:58:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:58:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:58:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:58:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:58:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:58:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:58:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 486 2024-09-21T23:58:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 488 2024-09-21T23:58:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:58:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:58:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:58:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:58:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:58:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:58:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 487 2024-09-21T23:58:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 489 2024-09-21T23:58:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:58:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:58:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:58:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:58:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:58:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:58:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 491 2024-09-21T23:58:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 490 2024-09-21T23:58:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:58:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:58:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:58:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:58:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:58:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:58:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:58:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:58:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:58:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:58:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 493 2024-09-21T23:59:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 492 2024-09-21T23:59:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:59:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:59:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:59:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:59:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:59:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:59:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:59:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:59:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:59:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:59:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:59:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:59:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 495 2024-09-21T23:59:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 494 2024-09-21T23:59:01.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 497 2024-09-21T23:59:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 496 2024-09-21T23:59:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:59:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:59:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:59:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:02.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:02.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:59:02.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:59:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:59:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 499 2024-09-21T23:59:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 498 2024-09-21T23:59:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:59:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:59:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:59:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:59:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:59:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:59:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 500 2024-09-21T23:59:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:59:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:59:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:59:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:59:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:59:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:59:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 501 2024-09-21T23:59:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 502 2024-09-21T23:59:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 503 2024-09-21T23:59:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:59:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:59:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:59:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:59:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:59:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:59:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:59:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:59:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:59:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:59:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:59:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:59:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 504 2024-09-21T23:59:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 505 2024-09-21T23:59:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 507 2024-09-21T23:59:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:59:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:59:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:59:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 506 2024-09-21T23:59:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:59:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:59:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:59:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 509 2024-09-21T23:59:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 508 2024-09-21T23:59:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:59:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:59:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:59:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:59:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:59:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:59:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 510 2024-09-21T23:59:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 511 2024-09-21T23:59:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:59:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:59:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:59:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:59:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:59:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:59:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 513 2024-09-21T23:59:10.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 512 2024-09-21T23:59:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:59:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:59:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:59:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:59:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:59:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:59:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 515 2024-09-21T23:59:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 514 2024-09-21T23:59:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:59:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:59:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:59:11.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:59:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:59:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:59:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 517 2024-09-21T23:59:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 516 2024-09-21T23:59:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:59:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:59:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:59:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:59:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:59:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:59:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:12.839 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:59:12.835+0000 7f54f2d95640 -1 reset not still connected to 0x5617fb678000 2024-09-21T23:59:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 518 2024-09-21T23:59:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 519 2024-09-21T23:59:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:59:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:59:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:59:13.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:13.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:59:13.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:59:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:59:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 521 2024-09-21T23:59:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 520 2024-09-21T23:59:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:59:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:59:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:59:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:59:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:59:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:59:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 523 2024-09-21T23:59:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 522 2024-09-21T23:59:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:59:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:59:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:59:15.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:59:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:59:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:59:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 524 2024-09-21T23:59:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 525 2024-09-21T23:59:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:59:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:59:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:59:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:59:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:59:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:59:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:17.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 527 2024-09-21T23:59:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:59:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:59:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:59:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 526 2024-09-21T23:59:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:59:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:59:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:59:17.894 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-21T23:59:17.895+0000 7fab5fe19640 -1 reset not still connected to 0x5623fc4bc340 2024-09-21T23:59:17.895 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-21T23:59:17.895+0000 7fab5fe19640 -1 reset not still connected to 0x5623fe5a4a90 2024-09-21T23:59:17.895 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-21T23:59:17.895+0000 7fab5fe19640 -1 reset not still connected to 0x56241ac78820 2024-09-21T23:59:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 529 2024-09-21T23:59:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 528 2024-09-21T23:59:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:59:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:59:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:59:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:59:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:59:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:59:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 531 2024-09-21T23:59:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 530 2024-09-21T23:59:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:59:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:59:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:59:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:59:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:59:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:59:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 533 2024-09-21T23:59:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 532 2024-09-21T23:59:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:59:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:59:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:59:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:59:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:59:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:59:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:59:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:59:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:59:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:59:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:59:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:59:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 534 2024-09-21T23:59:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 535 2024-09-21T23:59:21.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 536 2024-09-21T23:59:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:59:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:59:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:59:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 537 2024-09-21T23:59:22.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:22.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:59:22.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:22.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:22.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:59:22.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:59:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 538 2024-09-21T23:59:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 539 2024-09-21T23:59:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:59:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:59:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:59:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:59:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:59:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:59:24.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 540 2024-09-21T23:59:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 541 2024-09-21T23:59:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:59:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:59:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:59:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:59:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:59:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:59:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 543 2024-09-21T23:59:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 542 2024-09-21T23:59:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:59:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:59:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:59:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:59:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:59:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:59:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:59:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:59:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:59:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:59:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:59:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:59:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 544 2024-09-21T23:59:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 545 2024-09-21T23:59:26.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:26.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 546 2024-09-21T23:59:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:59:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:59:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:59:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:59:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:59:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:59:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 547 2024-09-21T23:59:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 548 2024-09-21T23:59:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 549 2024-09-21T23:59:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:59:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:59:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:59:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:59:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:59:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:59:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 551 2024-09-21T23:59:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 550 2024-09-21T23:59:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:59:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:59:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:59:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:59:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:59:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:59:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 552 2024-09-21T23:59:30.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 553 2024-09-21T23:59:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:59:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:59:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:59:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:59:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:59:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:59:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:59:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:59:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:59:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:59:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:59:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:59:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 554 2024-09-21T23:59:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 555 2024-09-21T23:59:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 556 2024-09-21T23:59:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:59:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:59:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:59:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:59:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:59:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:59:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 557 2024-09-21T23:59:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 558 2024-09-21T23:59:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 559 2024-09-21T23:59:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:59:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:59:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:59:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:59:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:59:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:59:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 560 2024-09-21T23:59:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:59:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:59:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:59:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:59:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:59:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:59:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 562 2024-09-21T23:59:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 561 2024-09-21T23:59:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:59:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:59:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:59:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 563 2024-09-21T23:59:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:59:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:59:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:59:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:36.518 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:59:36.519+0000 7f54f2d95640 -1 reset not still connected to 0x5617ffad5380 2024-09-21T23:59:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:59:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:59:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:59:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:59:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:59:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:59:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 565 2024-09-21T23:59:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 564 2024-09-21T23:59:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 566 2024-09-21T23:59:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 567 2024-09-21T23:59:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:59:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:59:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:59:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 569 2024-09-21T23:59:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 568 2024-09-21T23:59:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:59:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:59:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:59:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:59:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:59:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:59:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:59:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:59:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:59:39.763 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-21T23:59:39.755+0000 7fab1f496640 -1 reset not still connected to 0x55fa63cf7930 2024-09-21T23:59:39.763 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-21T23:59:39.755+0000 7fab1f496640 -1 reset not still connected to 0x55fa63f40270 2024-09-21T23:59:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 570 2024-09-21T23:59:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 571 2024-09-21T23:59:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:59:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:59:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:59:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:59:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:59:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:59:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 572 2024-09-21T23:59:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 573 2024-09-21T23:59:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:59:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:59:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:59:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:59:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:59:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:59:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 574 2024-09-21T23:59:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 575 2024-09-21T23:59:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:59:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:59:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:59:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:59:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:59:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:59:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 576 2024-09-21T23:59:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 577 2024-09-21T23:59:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:59:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:59:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:59:42.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:59:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:59:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:59:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:43.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 579 2024-09-21T23:59:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 578 2024-09-21T23:59:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:59:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:59:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:59:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:59:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:59:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:59:43.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 580 2024-09-21T23:59:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 581 2024-09-21T23:59:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:59:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:59:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:59:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:59:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:59:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:59:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 582 2024-09-21T23:59:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:59:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:59:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:59:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 583 2024-09-21T23:59:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:59:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:59:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:59:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 584 2024-09-21T23:59:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:59:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:59:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:59:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:59:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:59:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:59:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 585 2024-09-21T23:59:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 587 2024-09-21T23:59:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:59:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:59:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:59:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:59:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:59:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:59:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 586 2024-09-21T23:59:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 588 2024-09-21T23:59:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 589 2024-09-21T23:59:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:59:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:59:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:59:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:59:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:59:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:59:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 591 2024-09-21T23:59:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 590 2024-09-21T23:59:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:59:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:59:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:59:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:59:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:59:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:59:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:59:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:59:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:59:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:59:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:59:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:59:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 592 2024-09-21T23:59:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 593 2024-09-21T23:59:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:59:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:59:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:59:51.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 595 2024-09-21T23:59:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:59:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:59:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:59:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 594 2024-09-21T23:59:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 597 2024-09-21T23:59:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:59:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:59:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:59:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 596 2024-09-21T23:59:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:59:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:59:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:59:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 598 2024-09-21T23:59:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 599 2024-09-21T23:59:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:59:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:59:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:59:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:59:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:59:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:59:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 601 2024-09-21T23:59:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 600 2024-09-21T23:59:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:59:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:59:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:59:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:59:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:59:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:59:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 603 2024-09-21T23:59:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 602 2024-09-21T23:59:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-21T23:59:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-21T23:59:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-21T23:59:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-21T23:59:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-21T23:59:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-21T23:59:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:55.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-21T23:59:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-21T23:59:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-21T23:59:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-21T23:59:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-21T23:59:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-21T23:59:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 604 2024-09-21T23:59:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 605 2024-09-21T23:59:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:57.044 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:59:57.042+0000 7f54f2d95640 -1 reset not still connected to 0x5617f7c6ca90 2024-09-21T23:59:57.045 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-21T23:59:57.042+0000 7f54f2d95640 -1 reset not still connected to 0x5617fb742c30 2024-09-21T23:59:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 606 2024-09-21T23:59:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-21T23:59:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-21T23:59:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-21T23:59:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 607 2024-09-21T23:59:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-21T23:59:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:57.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-21T23:59:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-21T23:59:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 609 2024-09-21T23:59:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 608 2024-09-21T23:59:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-21T23:59:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-21T23:59:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-21T23:59:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-21T23:59:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-21T23:59:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-21T23:59:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 611 2024-09-21T23:59:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 610 2024-09-21T23:59:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-21T23:59:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-21T23:59:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-21T23:59:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-21T23:59:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-21T23:59:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-21T23:59:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-21T23:59:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-21T23:59:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-21T23:59:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-21T23:59:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 612 2024-09-22T00:00:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 614 2024-09-22T00:00:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:00:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:00:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:00:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:00:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:00:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:00:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:00:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:00:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:00:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:00:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:00:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:00:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 615 2024-09-22T00:00:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 616 2024-09-22T00:00:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 617 2024-09-22T00:00:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 618 2024-09-22T00:00:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:00:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:00:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:00:02.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:02.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:00:02.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:02.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:00:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:00:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 619 2024-09-22T00:00:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 620 2024-09-22T00:00:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:00:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:00:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:00:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:00:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:00:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:00:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 621 2024-09-22T00:00:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 622 2024-09-22T00:00:04.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:00:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:00:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:00:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:00:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:00:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:00:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 623 2024-09-22T00:00:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 624 2024-09-22T00:00:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:00:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:00:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:00:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:00:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:00:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:00:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 626 2024-09-22T00:00:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 625 2024-09-22T00:00:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:00:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:00:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:00:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:00:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:00:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:00:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 627 2024-09-22T00:00:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 628 2024-09-22T00:00:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:00:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:00:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:00:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:00:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:00:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:00:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 630 2024-09-22T00:00:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 629 2024-09-22T00:00:08.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:00:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:00:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:00:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:00:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:00:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:00:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 631 2024-09-22T00:00:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 632 2024-09-22T00:00:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:00:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:00:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:00:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:00:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:00:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:00:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 633 2024-09-22T00:00:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 634 2024-09-22T00:00:10.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:10.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:00:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:00:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:00:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:10.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:10.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:00:10.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:10.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:10.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:00:10.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:00:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 635 2024-09-22T00:00:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 636 2024-09-22T00:00:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:00:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:00:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:00:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:00:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 42197 2024-09-22T00:00:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 638 2024-09-22T00:00:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:00:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:00:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:00:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:00:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 42329 2024-09-22T00:00:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 637 2024-09-22T00:00:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 639 2024-09-22T00:00:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:00:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:00:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:00:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:00:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 42456 2024-09-22T00:00:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 640 2024-09-22T00:00:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:00:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:00:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:00:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:00:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 42603 2024-09-22T00:00:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 641 2024-09-22T00:00:15.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:00:15.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 42750 2024-09-22T00:00:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:00:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:00:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:00:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:00:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:00:16.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:00:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:00:16.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 42898 2024-09-22T00:00:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 642 2024-09-22T00:00:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 643 2024-09-22T00:00:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:00:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:00:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 43046 2024-09-22T00:00:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:00:18.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 43194 2024-09-22T00:00:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:00:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 43362 2024-09-22T00:00:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:00:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:00:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 42076 2024-09-22T00:02:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:02:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:02:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:02:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 42198 2024-09-22T00:02:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:02:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:02:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:02:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 42330 2024-09-22T00:02:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:02:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:02:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:02:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 42457 2024-09-22T00:02:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:02:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:02:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:02:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 42604 2024-09-22T00:02:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:02:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:02:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:02:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 42751 2024-09-22T00:02:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:02:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:02:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:02:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 42899 2024-09-22T00:02:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:02:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:02:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:02:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 43047 2024-09-22T00:02:50.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:02:50.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:02:50.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:02:50.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 43195 2024-09-22T00:02:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:02:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:02:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:02:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 43363 2024-09-22T00:02:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:02:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:02:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:02:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:02:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:02:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:02:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 1 2024-09-22T00:02:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:02:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd0 2024-09-22T00:02:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:02:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary1 2024-09-22T00:02:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary1 2024-09-22T00:02:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:02:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary1 2024-09-22T00:02:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 2 2024-09-22T00:02:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:02:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd1 2024-09-22T00:02:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 3 2024-09-22T00:02:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary2 2024-09-22T00:02:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:02:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary2 2024-09-22T00:02:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:02:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary2 2024-09-22T00:02:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd2 2024-09-22T00:02:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary3 2024-09-22T00:02:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary3 2024-09-22T00:02:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:02:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary3 2024-09-22T00:02:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:02:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:02:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 4 2024-09-22T00:02:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd3 2024-09-22T00:02:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary4 2024-09-22T00:02:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:02:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 5 2024-09-22T00:02:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary4 2024-09-22T00:02:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:02:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary4 2024-09-22T00:02:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:02:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd4 2024-09-22T00:02:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary5 2024-09-22T00:02:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary5 2024-09-22T00:02:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:02:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary5 2024-09-22T00:02:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:02:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:02:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 6 2024-09-22T00:02:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd5 2024-09-22T00:02:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary6 2024-09-22T00:02:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary6 2024-09-22T00:02:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:02:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary6 2024-09-22T00:02:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:02:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:02:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 7 2024-09-22T00:02:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd6 2024-09-22T00:02:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary7 2024-09-22T00:02:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary7 2024-09-22T00:02:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:02:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:02:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary7 2024-09-22T00:02:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:02:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:02:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:02:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:02:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:02:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 49563 2024-09-22T00:02:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 10 2024-09-22T00:02:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd9 2024-09-22T00:02:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary10 2024-09-22T00:02:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary10 2024-09-22T00:02:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:02:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary10 2024-09-22T00:02:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 9 2024-09-22T00:02:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd8 2024-09-22T00:02:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary9 2024-09-22T00:02:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary9 2024-09-22T00:02:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:02:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary9 2024-09-22T00:02:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 8 2024-09-22T00:02:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd7 2024-09-22T00:02:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary8 2024-09-22T00:02:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary8 2024-09-22T00:02:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:02:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary8 2024-09-22T00:02:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:02:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-22T00:02:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-22T00:02:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:02:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-22T00:02:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:02:53.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-22T00:02:53.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:02:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:02:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-22T00:02:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-22T00:02:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:02:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-22T00:02:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:02:53.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-22T00:02:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:02:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-22T00:02:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-22T00:02:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:02:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:03:24.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=bf257358ae085331ab151f9849dda0be 2024-09-22T00:03:24.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-22T00:03:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=a91c5b0c0baeff6af2a7661e86c229f1 2024-09-22T00:03:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-22T00:03:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=370f359b520748587286feacf52890a1 2024-09-22T00:03:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-22T00:03:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-22T00:03:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:03:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-22T00:03:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-22T00:03:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:03:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-22T00:03:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-22T00:03:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:03:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-22T00:03:25.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=07cf24e8bca50c4d220796ffd1c83dab 2024-09-22T00:03:25.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-22T00:03:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=0fab2ff495136c7aedb7182b619023e3 2024-09-22T00:03:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-22T00:03:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-22T00:03:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:03:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-22T00:03:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-22T00:03:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:03:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-22T00:03:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=ff8d8d9b7889d475e233aadb4c2a89fa 2024-09-22T00:03:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-22T00:03:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-22T00:03:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:03:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-22T00:03:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=684e570a5278953d20efafd008914447 2024-09-22T00:03:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-22T00:03:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-22T00:03:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:03:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-22T00:03:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=8135f9b8e6a424b50e047325204f2c70 2024-09-22T00:03:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-22T00:03:27.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=0b42d5b3837db6fc2d26f68e4ad6bdda 2024-09-22T00:03:27.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-22T00:03:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:03:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:03:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=95366ed186e6e5686325797a7b08c742 2024-09-22T00:03:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-22T00:03:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-22T00:03:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:03:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-22T00:03:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:03:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:03:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:03:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-22T00:03:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:03:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-22T00:03:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:03:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-22T00:03:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:03:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-22T00:03:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:03:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-22T00:03:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:03:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-22T00:03:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:03:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-22T00:03:27.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-22T00:03:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:03:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-22T00:03:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:03:28.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:03:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-22T00:03:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:03:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-22T00:03:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:03:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-22T00:03:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:03:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:03:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-22T00:03:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:03:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:03:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-22T00:03:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:03:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:03:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-22T00:03:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:03:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:03:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-22T00:03:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:03:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:03:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:03:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e7e01dd6-909d-415a-85aa-040f7cd3f42c 2024-09-22T00:03:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:03:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:03:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:25 2024-09-22T00:03:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:03:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:27' 2024-09-22T00:03:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:03:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 75eb3074-7ec8-41d0-818c-5d73fe84dc99 2024-09-22T00:03:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:03:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:03:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:25 2024-09-22T00:03:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:03:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:27' 2024-09-22T00:03:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:03:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 74559236-7ec4-4cda-b42c-9d9866ab7ec4 2024-09-22T00:03:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:03:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:03:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:25 2024-09-22T00:03:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:03:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:27' 2024-09-22T00:03:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:03:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:03:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:03:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:03:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:03:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:03:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:03:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: a97fee7f-8f8d-4ae7-8828-dad7b24deda0 2024-09-22T00:03:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:03:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:03:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:25 2024-09-22T00:03:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:03:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:27' 2024-09-22T00:03:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:03:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:03:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:03:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 99b83aef-02ad-4e23-8d94-f5c1d21275b6 2024-09-22T00:03:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:03:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:03:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:25 2024-09-22T00:03:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:03:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:27' 2024-09-22T00:03:28.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:03:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:03:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:03:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:03:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:03:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-22T00:03:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:03:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-22T00:03:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:03:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-22T00:03:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:03:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:03:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-22T00:03:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:03:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-22T00:03:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:03:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:03:29.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:03:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 007a3581-8ac5-40d4-afd3-4c2a0da39a63 2024-09-22T00:03:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:03:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:03:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:25 2024-09-22T00:03:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:03:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:28' 2024-09-22T00:03:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:03:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:03:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:03:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 2d731e2c-10e1-40be-a168-8f60f68f0436 2024-09-22T00:03:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:03:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:03:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:25 2024-09-22T00:03:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:03:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:28' 2024-09-22T00:03:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:03:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:03:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-22T00:03:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-22T00:03:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:03:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:03:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:03:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:03:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-22T00:03:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:03:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:03:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:03:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 2af3188f-f478-4e84-ab55-bc009d9f76ef 2024-09-22T00:03:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:03:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:03:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:25 2024-09-22T00:03:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:03:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:29' 2024-09-22T00:03:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:03:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:03:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:03:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8c18f283-4807-40f7-a279-3c54d0de9982 2024-09-22T00:03:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:03:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:03:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:25 2024-09-22T00:03:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:03:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:29' 2024-09-22T00:03:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:03:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:03:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:03:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e0332846-87f6-41a2-87e5-407f3baf7638 2024-09-22T00:03:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:03:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:03:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:25 2024-09-22T00:03:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:03:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:29' 2024-09-22T00:03:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:03:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:03:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-22T00:03:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-22T00:03:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-22T00:03:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:03:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:03:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:03:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:03:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:03:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:03:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-22T00:03:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:03:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:03:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-22T00:03:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:03:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:03:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:03:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e7e01dd6-909d-415a-85aa-040f7cd3f42c 2024-09-22T00:03:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:03:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:27' 2024-09-22T00:03:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 74559236-7ec4-4cda-b42c-9d9866ab7ec4 2024-09-22T00:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:27' 2024-09-22T00:03:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:03:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 75eb3074-7ec8-41d0-818c-5d73fe84dc99 2024-09-22T00:03:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30' 2024-09-22T00:03:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:03:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: a97fee7f-8f8d-4ae7-8828-dad7b24deda0 2024-09-22T00:03:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30' 2024-09-22T00:03:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:03:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:03:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:03:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:03:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:03:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 99b83aef-02ad-4e23-8d94-f5c1d21275b6 2024-09-22T00:03:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30' 2024-09-22T00:03:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:03:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:03:30.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:03:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:03:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:03:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:03:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:03:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:03:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:03:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:03:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:03:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-22T00:03:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:03:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-22T00:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-22T00:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-22T00:03:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:03:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-22T00:03:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:03:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:31.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-22T00:03:31.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:31.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:03:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:03:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-22T00:03:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:03:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:03:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:03:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 007a3581-8ac5-40d4-afd3-4c2a0da39a63 2024-09-22T00:03:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30' 2024-09-22T00:03:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:03:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 2d731e2c-10e1-40be-a168-8f60f68f0436 2024-09-22T00:03:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30' 2024-09-22T00:03:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:03:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:03:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:03:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:03:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-22T00:03:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:03:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:03:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:03:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-22T00:03:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:03:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-22T00:03:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-22T00:03:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-22T00:03:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:03:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-22T00:03:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:03:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:03:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:03:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-22T00:03:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:03:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-22T00:03:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-22T00:03:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-22T00:03:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-22T00:03:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-22T00:03:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:03:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e7e01dd6-909d-415a-85aa-040f7cd3f42c 2024-09-22T00:03:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:03:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30' 2024-09-22T00:03:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:03:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 99b83aef-02ad-4e23-8d94-f5c1d21275b6 2024-09-22T00:03:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:03:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30' 2024-09-22T00:03:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:03:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 74559236-7ec4-4cda-b42c-9d9866ab7ec4 2024-09-22T00:03:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:03:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30' 2024-09-22T00:03:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:03:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: a97fee7f-8f8d-4ae7-8828-dad7b24deda0 2024-09-22T00:03:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:03:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30' 2024-09-22T00:03:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:31.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:31.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:31.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:31.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:03:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 75eb3074-7ec8-41d0-818c-5d73fe84dc99 2024-09-22T00:03:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:03:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30' 2024-09-22T00:03:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:03:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:03:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:03:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:03:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:03:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:03:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:03:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:03:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-22T00:03:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:03:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:03:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-22T00:03:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-22T00:03:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:03:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:03:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-22T00:03:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:03:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:03:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-22T00:03:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:03:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:03:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-22T00:03:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:03:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:03:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:03:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-22T00:03:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:03:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:03:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-22T00:03:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:03:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-22T00:03:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:03:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:03:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-22T00:03:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-22T00:03:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:03:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:03:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-22T00:03:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:03:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:03:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-22T00:03:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:03:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:03:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:03:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 2af3188f-f478-4e84-ab55-bc009d9f76ef 2024-09-22T00:03:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30' 2024-09-22T00:03:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:03:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8c18f283-4807-40f7-a279-3c54d0de9982 2024-09-22T00:03:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30' 2024-09-22T00:03:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:03:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:32.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:03:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:03:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e0332846-87f6-41a2-87e5-407f3baf7638 2024-09-22T00:03:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:03:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30' 2024-09-22T00:03:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:03:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:03:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-22T00:03:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:03:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-22T00:03:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:03:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:03:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:03:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:03:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-22T00:03:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:03:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-22T00:03:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:03:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:32.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:03:30cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:32.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:32.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:03:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:03:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-22T00:03:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:03:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:03:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:03:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:03:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 007a3581-8ac5-40d4-afd3-4c2a0da39a63 2024-09-22T00:03:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:03:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30' 2024-09-22T00:03:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:32.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:32.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:32.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:32.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:32.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:03:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:03:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:03:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:03:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:03:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:03:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-22T00:03:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:03:32.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:03:32.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:03:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:03:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-22T00:03:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:03:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:03:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-22T00:03:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:03:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:03:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-22T00:03:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-22T00:03:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:03:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-22T00:03:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:03:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:03:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-22T00:03:32.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:32.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:32.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:03:32.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:32.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:03:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-22T00:03:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:03:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:03:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-22T00:03:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:03:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 2d731e2c-10e1-40be-a168-8f60f68f0436 2024-09-22T00:03:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:03:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30' 2024-09-22T00:03:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd0 2024-09-22T00:03:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-22T00:03:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:03:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd2 2024-09-22T00:03:32.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:32.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:32.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-22T00:03:32.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:32.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:32.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:03:32.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:32.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:03:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd1 2024-09-22T00:03:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-22T00:03:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:03:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd3 2024-09-22T00:03:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-22T00:03:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd4 2024-09-22T00:03:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:03:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:03:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-22T00:03:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:03:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:03:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-22T00:03:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:03:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:03:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-22T00:03:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-22T00:03:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:03:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-22T00:03:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-22T00:03:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:03:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-22T00:03:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-22T00:03:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:03:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:03:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:03:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:33.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-22T00:03:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:03:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8c18f283-4807-40f7-a279-3c54d0de9982 2024-09-22T00:03:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:03:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30' 2024-09-22T00:03:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:03:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 2af3188f-f478-4e84-ab55-bc009d9f76ef 2024-09-22T00:03:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:03:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30' 2024-09-22T00:03:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:03:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:03:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e0332846-87f6-41a2-87e5-407f3baf7638 2024-09-22T00:03:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:03:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30 2024-09-22T00:03:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:03:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:03:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:03:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:03:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:03:30' 2024-09-22T00:03:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:03:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:03:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:03:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:03:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:03:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:03:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:03:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:03:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:03:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:03:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:03:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-22 00:03:30' 2024-09-22T00:03:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:03:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:03:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:03:30' 2024-09-22T00:03:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:03:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:03:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:03:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:03:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:03:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:03:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:03:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-22T00:03:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:03:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:03:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-22T00:03:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:33.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:03:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:03:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:03:33.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:33.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:03:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-22T00:03:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:03:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:03:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-22T00:03:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-22T00:03:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:03:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:03:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-22T00:03:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:03:30cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:03:30' 2024-09-22T00:03:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:03:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:03:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:03:30' 2024-09-22T00:03:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:03:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:03:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:03:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:03:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:03:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:03:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-22T00:03:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:03:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:03:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:03:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:03:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-22T00:03:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd5 2024-09-22T00:03:34.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:03:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-22T00:03:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd6 2024-09-22T00:03:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:03:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-22T00:03:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:03:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:03:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:03:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:03:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:03:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:03:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:03:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-22T00:03:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:03:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:03:34.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-22T00:03:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-22T00:03:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd7 2024-09-22T00:03:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-22T00:03:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:03:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd8 2024-09-22T00:03:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-22T00:03:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:03:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd9 2024-09-22T00:03:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-22T00:03:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:04:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=0fab2ff495136c7aedb7182b619023e3 2024-09-22T00:04:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-22T00:04:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=bf257358ae085331ab151f9849dda0be 2024-09-22T00:04:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-22T00:04:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 0fab2ff495136c7aedb7182b619023e3 != \0\f\a\b\2\f\f\4\9\5\1\3\6\c\7\a\e\d\b\7\1\8\2\b\6\1\9\0\2\3\e\3 ]] 2024-09-22T00:04:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ bf257358ae085331ab151f9849dda0be != \b\f\2\5\7\3\5\8\a\e\0\8\5\3\3\1\a\b\1\5\1\f\9\8\4\9\d\d\a\0\b\e ]] 2024-09-22T00:04:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:04:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 49564 2024-09-22T00:04:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=07cf24e8bca50c4d220796ffd1c83dab 2024-09-22T00:04:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-22T00:04:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 07cf24e8bca50c4d220796ffd1c83dab != \0\7\c\f\2\4\e\8\b\c\a\5\0\c\4\d\2\2\0\7\9\6\f\f\d\1\c\8\3\d\a\b ]] 2024-09-22T00:04:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=370f359b520748587286feacf52890a1 2024-09-22T00:04:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-22T00:04:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 370f359b520748587286feacf52890a1 != \3\7\0\f\3\5\9\b\5\2\0\7\4\8\5\8\7\2\8\6\f\e\a\c\f\5\2\8\9\0\a\1 ]] 2024-09-22T00:04:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=a91c5b0c0baeff6af2a7661e86c229f1 2024-09-22T00:04:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-22T00:04:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=684e570a5278953d20efafd008914447 2024-09-22T00:04:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-22T00:04:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ a91c5b0c0baeff6af2a7661e86c229f1 != \a\9\1\c\5\b\0\c\0\b\a\e\f\f\6\a\f\2\a\7\6\6\1\e\8\6\c\2\2\9\f\1 ]] 2024-09-22T00:04:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 684e570a5278953d20efafd008914447 != \6\8\4\e\5\7\0\a\5\2\7\8\9\5\3\d\2\0\e\f\a\f\d\0\0\8\9\1\4\4\4\7 ]] 2024-09-22T00:04:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:04:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 49565 2024-09-22T00:04:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:04:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 49567 2024-09-22T00:04:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:04:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 49570 2024-09-22T00:04:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=ff8d8d9b7889d475e233aadb4c2a89fa 2024-09-22T00:04:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-22T00:04:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ ff8d8d9b7889d475e233aadb4c2a89fa != \f\f\8\d\8\d\9\b\7\8\8\9\d\4\7\5\e\2\3\3\a\a\d\b\4\c\2\a\8\9\f\a ]] 2024-09-22T00:04:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=0b42d5b3837db6fc2d26f68e4ad6bdda 2024-09-22T00:04:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-22T00:04:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 0b42d5b3837db6fc2d26f68e4ad6bdda != \0\b\4\2\d\5\b\3\8\3\7\d\b\6\f\c\2\d\2\6\f\6\8\e\4\a\d\6\b\d\d\a ]] 2024-09-22T00:04:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=8135f9b8e6a424b50e047325204f2c70 2024-09-22T00:04:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-22T00:04:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 8135f9b8e6a424b50e047325204f2c70 != \8\1\3\5\f\9\b\8\e\6\a\4\2\4\b\5\0\e\0\4\7\3\2\5\2\0\4\f\2\c\7\0 ]] 2024-09-22T00:04:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:04:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 49572 2024-09-22T00:04:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:04:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 49574 2024-09-22T00:04:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:04:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 49576 2024-09-22T00:04:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=95366ed186e6e5686325797a7b08c742 2024-09-22T00:04:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-22T00:04:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 95366ed186e6e5686325797a7b08c742 != \9\5\3\6\6\e\d\1\8\6\e\6\e\5\6\8\6\3\2\5\7\9\7\a\7\b\0\8\c\7\4\2 ]] 2024-09-22T00:04:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:04:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 49578 2024-09-22T00:04:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:04:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 49579 2024-09-22T00:04:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:04:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-22T00:04:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-22T00:04:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:04:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:04:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-22T00:04:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:04:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:04:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-22T00:04:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:04:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:04:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:04:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 75eb3074-7ec8-41d0-818c-5d73fe84dc99 2024-09-22T00:04:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:04:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3606528.0,"bytes_per_snapshot":72130560.0,"last_snapshot_bytes":72130560,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"} 2024-09-22T00:04:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:04:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:05 2024-09-22T00:04:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:04:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:04:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:04:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:04:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:05' 2024-09-22T00:04:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:04:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:04:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:04:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3606528.0,"bytes_per_snapshot":72130560.0,"last_snapshot_bytes":72130560,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"} 2024-09-22T00:04:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:04:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:04:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:04:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:04:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:04:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:04:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:05' 2024-09-22T00:04:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-22 00:04:05' 2024-09-22T00:04:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:04:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:04:05' 2024-09-22T00:04:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:04:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary175eb3074-7ec8-41d0-818c-5d73fe84dc99up+replayingreplaying, {"bytes_per_second":3606528.0,"bytes_per_snapshot":72130560.0,"last_snapshot_bytes":72130560,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:05image-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+replayingreplaying, {"bytes_per_second":3360153.6,"bytes_per_snapshot":67203072.0,"last_snapshot_bytes":67203072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:05image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+replayingreplaying, {"bytes_per_second":3611238.4,"bytes_per_snapshot":72224768.0,"last_snapshot_bytes":72224768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963413,"remote_snapshot_timestamp":1726963413,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:05image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+replayingreplaying, {"bytes_per_second":3604684.8,"bytes_per_snapshot":72093696.0,"last_snapshot_bytes":72093696,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:05image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+replayingreplaying, {"bytes_per_second":3589529.6,"bytes_per_snapshot":71790592.0,"last_snapshot_bytes":71790592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:05image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+replayingreplaying, {"bytes_per_second":3609600.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:05image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+replayingreplaying, {"bytes_per_second":3608780.8,"bytes_per_snapshot":72175616.0,"last_snapshot_bytes":72175616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:05image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+replayingreplaying, {"bytes_per_second":3087564.8,"bytes_per_snapshot":61751296.0,"last_snapshot_bytes":61751296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:05image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+replayingreplaying, {"bytes_per_second":3168460.8,"bytes_per_snapshot":63369216.0,"last_snapshot_bytes":63369216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:05image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+replayingreplaying, {"bytes_per_second":3415449.6,"bytes_per_snapshot":68308992.0,"last_snapshot_bytes":68308992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963413,"remote_snapshot_timestamp":1726963413,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:05' 2024-09-22T00:04:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:04:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:04:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:05' 2024-09-22T00:04:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:04:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:04:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:04:10.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-22T00:04:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:04:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:04:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-22T00:04:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-22T00:04:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:04:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-22T00:04:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:04:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:04:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary1 2024-09-22T00:04:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary1 2024-09-22T00:04:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-22T00:04:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-22T00:04:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:04:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:04:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-22T00:04:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:04:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:04:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-22T00:04:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:04:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:04:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:04:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 2d731e2c-10e1-40be-a168-8f60f68f0436 2024-09-22T00:04:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:04:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3611238.4,"bytes_per_snapshot":72224768.0,"last_snapshot_bytes":72224768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963413,"remote_snapshot_timestamp":1726963413,"replay_state":"idle"} 2024-09-22T00:04:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:04:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:05 2024-09-22T00:04:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:04:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:04:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:04:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:04:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:10' 2024-09-22T00:04:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:04:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:04:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:04:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3611238.4,"bytes_per_snapshot":72224768.0,"last_snapshot_bytes":72224768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963413,"remote_snapshot_timestamp":1726963413,"replay_state":"idle"} 2024-09-22T00:04:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:04:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:04:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:04:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:04:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:04:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:04:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:05' 2024-09-22T00:04:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-22 00:04:05' 2024-09-22T00:04:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:04:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:04:05' 2024-09-22T00:04:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:04:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK942614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+replayingreplaying, {"bytes_per_second":3360153.6,"bytes_per_snapshot":67203072.0,"last_snapshot_bytes":67203072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:10image-primary22d731e2c-10e1-40be-a168-8f60f68f0436up+replayingreplaying, {"bytes_per_second":3611238.4,"bytes_per_snapshot":72224768.0,"last_snapshot_bytes":72224768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963413,"remote_snapshot_timestamp":1726963413,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:10image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+replayingreplaying, {"bytes_per_second":3604684.8,"bytes_per_snapshot":72093696.0,"last_snapshot_bytes":72093696,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:10image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+replayingreplaying, {"bytes_per_second":3589529.6,"bytes_per_snapshot":71790592.0,"last_snapshot_bytes":71790592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:10image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+replayingreplaying, {"bytes_per_second":3609600.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:10image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+replayingreplaying, {"bytes_per_second":3608780.8,"bytes_per_snapshot":72175616.0,"last_snapshot_bytes":72175616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:10image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+replayingreplaying, {"bytes_per_second":3087564.8,"bytes_per_snapshot":61751296.0,"last_snapshot_bytes":61751296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:10image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+replayingreplaying, {"bytes_per_second":3168460.8,"bytes_per_snapshot":63369216.0,"last_snapshot_bytes":63369216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:10image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+replayingreplaying, {"bytes_per_second":3415449.6,"bytes_per_snapshot":68308992.0,"last_snapshot_bytes":68308992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963413,"remote_snapshot_timestamp":1726963413,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:10' 2024-09-22T00:04:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:04:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:04:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:05' 2024-09-22T00:04:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:04:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:04:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:04:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-22T00:04:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:04:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:04:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-22T00:04:13.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-22T00:04:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-22T00:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:04:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary2 2024-09-22T00:04:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary2 2024-09-22T00:04:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-22T00:04:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-22T00:04:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:04:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:04:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-22T00:04:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:04:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:04:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-22T00:04:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:04:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:04:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:04:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: a97fee7f-8f8d-4ae7-8828-dad7b24deda0 2024-09-22T00:04:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:04:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3604684.8,"bytes_per_snapshot":72093696.0,"last_snapshot_bytes":72093696,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"} 2024-09-22T00:04:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:04:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:05 2024-09-22T00:04:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:04:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:04:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:04:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:04:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:15' 2024-09-22T00:04:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:04:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:04:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:04:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3604684.8,"bytes_per_snapshot":72093696.0,"last_snapshot_bytes":72093696,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"} 2024-09-22T00:04:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:04:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:04:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:04:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:04:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:04:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:04:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:05' 2024-09-22T00:04:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-22 00:04:05' 2024-09-22T00:04:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:04:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:04:05' 2024-09-22T00:04:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:04:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK842614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+replayingreplaying, {"bytes_per_second":3360153.6,"bytes_per_snapshot":67203072.0,"last_snapshot_bytes":67203072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:15image-primary3a97fee7f-8f8d-4ae7-8828-dad7b24deda0up+replayingreplaying, {"bytes_per_second":3604684.8,"bytes_per_snapshot":72093696.0,"last_snapshot_bytes":72093696,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:15image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+replayingreplaying, {"bytes_per_second":3589529.6,"bytes_per_snapshot":71790592.0,"last_snapshot_bytes":71790592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:15image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+replayingreplaying, {"bytes_per_second":3609600.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:15image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+replayingreplaying, {"bytes_per_second":3608780.8,"bytes_per_snapshot":72175616.0,"last_snapshot_bytes":72175616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:15image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+replayingreplaying, {"bytes_per_second":3087564.8,"bytes_per_snapshot":61751296.0,"last_snapshot_bytes":61751296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:15image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+replayingreplaying, {"bytes_per_second":3168460.8,"bytes_per_snapshot":63369216.0,"last_snapshot_bytes":63369216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:15image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+replayingreplaying, {"bytes_per_second":3415449.6,"bytes_per_snapshot":68308992.0,"last_snapshot_bytes":68308992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963413,"remote_snapshot_timestamp":1726963413,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:15' 2024-09-22T00:04:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:04:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:04:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:05' 2024-09-22T00:04:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:04:16.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:04:16.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:16.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:04:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-22T00:04:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:04:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:04:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-22T00:04:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-22T00:04:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:04:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-22T00:04:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:04:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:04:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary3 2024-09-22T00:04:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary3 2024-09-22T00:04:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-22T00:04:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-22T00:04:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:04:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:04:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-22T00:04:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:04:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:04:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-22T00:04:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:04:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:04:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:04:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 74559236-7ec4-4cda-b42c-9d9866ab7ec4 2024-09-22T00:04:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:04:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3589529.6,"bytes_per_snapshot":71790592.0,"last_snapshot_bytes":71790592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"} 2024-09-22T00:04:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:04:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:05 2024-09-22T00:04:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:04:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:04:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:04:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:04:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:15' 2024-09-22T00:04:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:04:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:04:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:04:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3589529.6,"bytes_per_snapshot":71790592.0,"last_snapshot_bytes":71790592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"} 2024-09-22T00:04:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:04:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:04:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:04:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:04:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:04:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:04:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:05' 2024-09-22T00:04:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-22 00:04:05' 2024-09-22T00:04:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:04:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:04:05' 2024-09-22T00:04:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:04:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK742614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+replayingreplaying, {"bytes_per_second":3360153.6,"bytes_per_snapshot":67203072.0,"last_snapshot_bytes":67203072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:15image-primary474559236-7ec4-4cda-b42c-9d9866ab7ec4up+replayingreplaying, {"bytes_per_second":3589529.6,"bytes_per_snapshot":71790592.0,"last_snapshot_bytes":71790592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:15image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+replayingreplaying, {"bytes_per_second":3609600.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:15image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+replayingreplaying, {"bytes_per_second":3608780.8,"bytes_per_snapshot":72175616.0,"last_snapshot_bytes":72175616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:15image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+replayingreplaying, {"bytes_per_second":3087564.8,"bytes_per_snapshot":61751296.0,"last_snapshot_bytes":61751296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:15image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+replayingreplaying, {"bytes_per_second":3168460.8,"bytes_per_snapshot":63369216.0,"last_snapshot_bytes":63369216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:15image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+replayingreplaying, {"bytes_per_second":3415449.6,"bytes_per_snapshot":68308992.0,"last_snapshot_bytes":68308992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963413,"remote_snapshot_timestamp":1726963413,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:15' 2024-09-22T00:04:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:04:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:04:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:05' 2024-09-22T00:04:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:04:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:04:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:04:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-22T00:04:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:04:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:04:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-22T00:04:19.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-22T00:04:21.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:04:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-22T00:04:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:04:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:04:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary4 2024-09-22T00:04:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary4 2024-09-22T00:04:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-22T00:04:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-22T00:04:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:04:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:04:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-22T00:04:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:04:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:04:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-22T00:04:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:04:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e0332846-87f6-41a2-87e5-407f3baf7638 2024-09-22T00:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3609600.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"} 2024-09-22T00:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:05 2024-09-22T00:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:04:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:04:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:20' 2024-09-22T00:04:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:04:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:04:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:04:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3609600.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"} 2024-09-22T00:04:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:04:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:04:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:04:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:04:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:04:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:04:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:05' 2024-09-22T00:04:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-22 00:04:05' 2024-09-22T00:04:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:04:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:04:05' 2024-09-22T00:04:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:04:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK642614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+replayingreplaying, {"bytes_per_second":3360153.6,"bytes_per_snapshot":67203072.0,"last_snapshot_bytes":67203072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:20image-primary5e0332846-87f6-41a2-87e5-407f3baf7638up+replayingreplaying, {"bytes_per_second":3609600.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:20image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+replayingreplaying, {"bytes_per_second":3608780.8,"bytes_per_snapshot":72175616.0,"last_snapshot_bytes":72175616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:20image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+replayingreplaying, {"bytes_per_second":3087564.8,"bytes_per_snapshot":61751296.0,"last_snapshot_bytes":61751296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:20image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+replayingreplaying, {"bytes_per_second":3168460.8,"bytes_per_snapshot":63369216.0,"last_snapshot_bytes":63369216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:20image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+replayingreplaying, {"bytes_per_second":3415449.6,"bytes_per_snapshot":68308992.0,"last_snapshot_bytes":68308992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963413,"remote_snapshot_timestamp":1726963413,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:20' 2024-09-22T00:04:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:04:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:04:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:05' 2024-09-22T00:04:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:04:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:04:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:04:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-22T00:04:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:04:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:04:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-22T00:04:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-22T00:04:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:04:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-22T00:04:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:04:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:04:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary5 2024-09-22T00:04:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary5 2024-09-22T00:04:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-22T00:04:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-22T00:04:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:04:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:04:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-22T00:04:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:04:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:04:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-22T00:04:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:04:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:04:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:04:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8c18f283-4807-40f7-a279-3c54d0de9982 2024-09-22T00:04:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:04:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3608780.8,"bytes_per_snapshot":72175616.0,"last_snapshot_bytes":72175616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"} 2024-09-22T00:04:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:04:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:05 2024-09-22T00:04:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:04:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:04:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:04:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:04:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:20' 2024-09-22T00:04:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:04:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:04:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:04:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3608780.8,"bytes_per_snapshot":72175616.0,"last_snapshot_bytes":72175616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"} 2024-09-22T00:04:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:04:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:04:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:04:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:04:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:04:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:04:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:05' 2024-09-22T00:04:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-22 00:04:05' 2024-09-22T00:04:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:04:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:04:05' 2024-09-22T00:04:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:04:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK542614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+replayingreplaying, {"bytes_per_second":3360153.6,"bytes_per_snapshot":67203072.0,"last_snapshot_bytes":67203072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:20image-primary68c18f283-4807-40f7-a279-3c54d0de9982up+replayingreplaying, {"bytes_per_second":3608780.8,"bytes_per_snapshot":72175616.0,"last_snapshot_bytes":72175616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:20image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+replayingreplaying, {"bytes_per_second":3087564.8,"bytes_per_snapshot":61751296.0,"last_snapshot_bytes":61751296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:20image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+replayingreplaying, {"bytes_per_second":3168460.8,"bytes_per_snapshot":63369216.0,"last_snapshot_bytes":63369216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:20image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+replayingreplaying, {"bytes_per_second":3415449.6,"bytes_per_snapshot":68308992.0,"last_snapshot_bytes":68308992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963413,"remote_snapshot_timestamp":1726963413,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:20' 2024-09-22T00:04:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:04:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:04:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:05' 2024-09-22T00:04:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:04:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:04:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:04:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-22T00:04:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:04:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:04:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-22T00:04:25.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-22T00:04:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:04:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-22T00:04:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:04:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:04:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary6 2024-09-22T00:04:27.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:27.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary6 2024-09-22T00:04:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-22T00:04:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-22T00:04:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:04:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:04:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-22T00:04:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:04:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:04:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-22T00:04:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:04:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:28.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:28.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:28.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:28.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:28.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:04:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:04:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e7e01dd6-909d-415a-85aa-040f7cd3f42c 2024-09-22T00:04:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:04:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3087564.8,"bytes_per_snapshot":61751296.0,"last_snapshot_bytes":61751296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"} 2024-09-22T00:04:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:04:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:05 2024-09-22T00:04:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:04:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:04:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:04:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:04:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:25' 2024-09-22T00:04:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:04:28.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:04:28.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:04:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3087564.8,"bytes_per_snapshot":61751296.0,"last_snapshot_bytes":61751296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"} 2024-09-22T00:04:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:04:28.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:28.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:04:28.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:04:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:04:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:04:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:04:28.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:05' 2024-09-22T00:04:28.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-22 00:04:05' 2024-09-22T00:04:28.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:04:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:04:05' 2024-09-22T00:04:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:04:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK442614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+replayingreplaying, {"bytes_per_second":3360153.6,"bytes_per_snapshot":67203072.0,"last_snapshot_bytes":67203072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:25image-primary7e7e01dd6-909d-415a-85aa-040f7cd3f42cup+replayingreplaying, {"bytes_per_second":3087564.8,"bytes_per_snapshot":61751296.0,"last_snapshot_bytes":61751296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:25image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+replayingreplaying, {"bytes_per_second":3168460.8,"bytes_per_snapshot":63369216.0,"last_snapshot_bytes":63369216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:25image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+replayingreplaying, {"bytes_per_second":3415449.6,"bytes_per_snapshot":68308992.0,"last_snapshot_bytes":68308992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963413,"remote_snapshot_timestamp":1726963413,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:25' 2024-09-22T00:04:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:04:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:04:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:05' 2024-09-22T00:04:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:04:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:04:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:04:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-22T00:04:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:04:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:04:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-22T00:04:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-22T00:04:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:04:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-22T00:04:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:04:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:04:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary7 2024-09-22T00:04:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary7 2024-09-22T00:04:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-22T00:04:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-22T00:04:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:04:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:04:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-22T00:04:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:04:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:04:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-22T00:04:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:04:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:04:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:04:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 2af3188f-f478-4e84-ab55-bc009d9f76ef 2024-09-22T00:04:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:04:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3168460.8,"bytes_per_snapshot":63369216.0,"last_snapshot_bytes":63369216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"} 2024-09-22T00:04:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:04:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:05 2024-09-22T00:04:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:04:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:04:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:04:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:04:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:30' 2024-09-22T00:04:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:04:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:04:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:04:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3168460.8,"bytes_per_snapshot":63369216.0,"last_snapshot_bytes":63369216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"} 2024-09-22T00:04:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:04:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:04:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:04:31.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:04:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:04:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:04:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:05' 2024-09-22T00:04:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-22 00:04:05' 2024-09-22T00:04:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:04:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:04:05' 2024-09-22T00:04:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:04:31.533 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:04:31.531+0000 7fab5fe19640 -1 reset not still connected to 0x562411925520 2024-09-22T00:04:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+replayingreplaying, {"bytes_per_second":3360153.6,"bytes_per_snapshot":67203072.0,"last_snapshot_bytes":67203072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:30image-primary82af3188f-f478-4e84-ab55-bc009d9f76efup+replayingreplaying, {"bytes_per_second":3168460.8,"bytes_per_snapshot":63369216.0,"last_snapshot_bytes":63369216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963414,"remote_snapshot_timestamp":1726963414,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+replayingreplaying, {"bytes_per_second":3415449.6,"bytes_per_snapshot":68308992.0,"last_snapshot_bytes":68308992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963413,"remote_snapshot_timestamp":1726963413,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:30' 2024-09-22T00:04:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:04:31.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:04:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:05' 2024-09-22T00:04:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:04:31.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:04:31.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:31.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:04:31.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-22T00:04:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:04:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:04:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-22T00:04:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-22T00:04:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:04:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-22T00:04:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:04:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:04:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary8 2024-09-22T00:04:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary8 2024-09-22T00:04:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-22T00:04:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-22T00:04:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:04:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:04:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-22T00:04:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:04:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:04:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-22T00:04:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:04:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:04:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:04:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 007a3581-8ac5-40d4-afd3-4c2a0da39a63 2024-09-22T00:04:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:04:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3415449.6,"bytes_per_snapshot":68308992.0,"last_snapshot_bytes":68308992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963413,"remote_snapshot_timestamp":1726963413,"replay_state":"idle"} 2024-09-22T00:04:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:04:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:05 2024-09-22T00:04:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:04:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:04:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:04:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:04:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:30' 2024-09-22T00:04:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:04:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:04:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:04:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3415449.6,"bytes_per_snapshot":68308992.0,"last_snapshot_bytes":68308992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963413,"remote_snapshot_timestamp":1726963413,"replay_state":"idle"} 2024-09-22T00:04:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:04:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:04:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:04:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:04:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:04:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:04:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:05' 2024-09-22T00:04:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-22 00:04:05' 2024-09-22T00:04:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:04:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:04:05' 2024-09-22T00:04:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:04:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK242614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+replayingreplaying, {"bytes_per_second":3360153.6,"bytes_per_snapshot":67203072.0,"last_snapshot_bytes":67203072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:30image-primary9007a3581-8ac5-40d4-afd3-4c2a0da39a63up+replayingreplaying, {"bytes_per_second":3415449.6,"bytes_per_snapshot":68308992.0,"last_snapshot_bytes":68308992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963413,"remote_snapshot_timestamp":1726963413,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:30' 2024-09-22T00:04:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:04:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:04:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:05' 2024-09-22T00:04:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:04:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:04:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:04:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-22T00:04:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:04:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:04:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-22T00:04:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-22T00:04:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:04:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-22T00:04:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:04:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:04:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary9 2024-09-22T00:04:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary9 2024-09-22T00:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-22T00:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-22T00:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-22T00:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:04:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:04:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-22T00:04:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:04:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:04:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:04:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:04:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:04:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 99b83aef-02ad-4e23-8d94-f5c1d21275b6 2024-09-22T00:04:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:04:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67203072.0,"last_snapshot_bytes":67203072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"} 2024-09-22T00:04:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:04:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:35 2024-09-22T00:04:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:04:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:04:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:04:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:04:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:04:35' 2024-09-22T00:04:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:04:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:04:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:04:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67203072.0,"last_snapshot_bytes":67203072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"} 2024-09-22T00:04:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:04:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:04:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:04:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:04:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:04:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:35' 2024-09-22T00:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-22 00:04:35' 2024-09-22T00:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:04:35' 2024-09-22T00:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:04:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:04:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:04:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK142614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1099b83aef-02ad-4e23-8d94-f5c1d21275b6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67203072.0,"last_snapshot_bytes":67203072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963412,"remote_snapshot_timestamp":1726963412,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:04:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:04:35' 2024-09-22T00:04:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:04:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:04:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:04:35' 2024-09-22T00:04:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:04:37.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:04:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:04:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:04:37.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:04:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-22T00:04:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:04:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:04:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-22T00:04:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-22T00:04:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:04:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-22T00:04:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:04:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:04:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:04:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary10 2024-09-22T00:04:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary10 2024-09-22T00:04:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..10} 2024-09-22T00:04:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS=() 2024-09-22T00:04:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS=() 2024-09-22T00:04:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS=() 2024-09-22T00:04:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS=() 2024-09-22T00:04:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ RET=0 2024-09-22T00:04:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-22T00:04:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary1 2024-09-22T00:04:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-22T00:04:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:04:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:04:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-22T00:04:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:04:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:04:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:04:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:04:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-22T00:04:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:04:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:04:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-22T00:04:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:04:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-22T00:04:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:04:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-22T00:04:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary1': 2024-09-22T00:04:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:04:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:04:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:04:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 19c07a04d1af 2024-09-22T00:04:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.19c07a04d1af 2024-09-22T00:04:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:04:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:04:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:04:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:04:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:04:39 2024 2024-09-22T00:04:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:04:39 2024 2024-09-22T00:04:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:04:39 2024 2024-09-22T00:04:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:04:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:04:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: ffe30d9d-5e74-4ba9-81a4-1f3439aea154 2024-09-22T00:04:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:04:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:04:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:04:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-22T00:04:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd0 2024-09-22T00:04:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:04:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-22T00:04:40.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:04:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:04:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:04:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: ae32865f-2cc9-4d91-812e-f06c605bc40b 2024-09-22T00:04:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:04:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:04:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:04:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:04:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:04:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:04:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary1 2024-09-22T00:04:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-22T00:04:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:04:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary1 2024-09-22T00:04:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:04:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-22T00:04:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:04:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:04:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:04:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-22T00:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary1 2024-09-22T00:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary2 2024-09-22T00:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-22T00:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary1 2024-09-22T00:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-22T00:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-22T00:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:04:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:04:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:04:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:04:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-22T00:04:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:04:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:04:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-22T00:04:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:04:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-22T00:04:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 665 2024-09-22T00:04:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:04:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-22T00:04:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary2': 2024-09-22T00:04:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:04:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:04:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:04:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 19cef6d52305 2024-09-22T00:04:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.19cef6d52305 2024-09-22T00:04:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:04:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:04:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:04:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:04:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:04:40 2024 2024-09-22T00:04:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:04:40 2024 2024-09-22T00:04:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:04:40 2024 2024-09-22T00:04:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:04:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:04:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 183dbf3b-282a-4c6d-9ed1-e9865ba1f6db 2024-09-22T00:04:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:04:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:04:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:04:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-22T00:04:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd1 2024-09-22T00:04:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:04:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-22T00:04:41.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:04:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:04:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:04:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 1ee41682-0c5c-4316-ae4f-9eac7726b3a2 2024-09-22T00:04:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:04:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:04:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:04:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:04:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:04:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:04:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary2 2024-09-22T00:04:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-22T00:04:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:04:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary2 2024-09-22T00:04:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:04:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-22T00:04:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:04:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:04:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:04:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:04:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:04:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary2 2024-09-22T00:04:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-22T00:04:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary2 2024-09-22T00:04:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary3 2024-09-22T00:04:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-22T00:04:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-22T00:04:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:04:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:04:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-22T00:04:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:04:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:04:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:04:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:04:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-22T00:04:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:04:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-22T00:04:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:04:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:04:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-22T00:04:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:04:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-22T00:04:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:04:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-22T00:04:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:04:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 667 2024-09-22T00:04:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-22T00:04:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary3': 2024-09-22T00:04:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:04:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:04:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:04:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 19ddc7bb92c7 2024-09-22T00:04:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.19ddc7bb92c7 2024-09-22T00:04:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:04:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:04:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:04:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:04:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:04:41 2024 2024-09-22T00:04:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:04:41 2024 2024-09-22T00:04:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:04:41 2024 2024-09-22T00:04:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:04:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:04:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 7d3beb22-c450-46bd-9c24-aa8184e646ef 2024-09-22T00:04:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:04:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:04:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:04:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-22T00:04:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd2 2024-09-22T00:04:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:04:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-22T00:04:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:04:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:04:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:04:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 6326f99a-ee8b-4bf7-b675-291e4da721bf 2024-09-22T00:04:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:04:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:04:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:04:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:04:42.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:04:42.257 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:04:42.259+0000 7fab5fe19640 -1 reset not still connected to 0x5623fe71bba0 2024-09-22T00:04:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:04:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary3 2024-09-22T00:04:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-22T00:04:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:04:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary3 2024-09-22T00:04:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:04:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-22T00:04:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:04:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary3 2024-09-22T00:04:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-22T00:04:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary4 2024-09-22T00:04:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:04:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-22T00:04:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:04:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary3 2024-09-22T00:04:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:04:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-22T00:04:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-22T00:04:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:04:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:04:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:04:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:04:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:04:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-22T00:04:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:04:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:04:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:04:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:04:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-22T00:04:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:04:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-22T00:04:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:04:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-22T00:04:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:04:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 669 2024-09-22T00:04:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-22T00:04:43.061 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:04:43.063+0000 7fab5fe19640 -1 reset not still connected to 0x5624088f6ea0 2024-09-22T00:04:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary4': 2024-09-22T00:04:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:04:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:04:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:04:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 19ea9294b2bd 2024-09-22T00:04:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.19ea9294b2bd 2024-09-22T00:04:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:04:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:04:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:04:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:04:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:04:42 2024 2024-09-22T00:04:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:04:42 2024 2024-09-22T00:04:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:04:42 2024 2024-09-22T00:04:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:04:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:04:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 859a2725-431b-4842-85c5-dceb1043514d 2024-09-22T00:04:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:04:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:04:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:04:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-22T00:04:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd3 2024-09-22T00:04:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:04:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-22T00:04:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:04:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:04:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:04:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 279b6ded-635f-4a0a-935a-fbfd8fa82f5a 2024-09-22T00:04:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:04:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:04:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:04:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:04:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:04:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:04:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary4 2024-09-22T00:04:43.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-22T00:04:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:04:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary4 2024-09-22T00:04:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:04:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-22T00:04:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:04:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:04:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-22T00:04:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary5 2024-09-22T00:04:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-22T00:04:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:04:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary4 2024-09-22T00:04:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:04:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:04:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:04:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:04:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-22T00:04:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary4 2024-09-22T00:04:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-22T00:04:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:04:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:04:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:04:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:04:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-22T00:04:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:04:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:04:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-22T00:04:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:04:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-22T00:04:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:04:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-22T00:04:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:04:43.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 671 2024-09-22T00:04:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-22T00:04:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary5': 2024-09-22T00:04:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:04:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:04:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:04:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 19f9fbd55714 2024-09-22T00:04:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.19f9fbd55714 2024-09-22T00:04:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:04:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:04:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:04:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:04:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:04:43 2024 2024-09-22T00:04:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:04:43 2024 2024-09-22T00:04:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:04:43 2024 2024-09-22T00:04:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:04:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:04:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 52725bae-bfe7-4465-ad3b-965fe6fc62de 2024-09-22T00:04:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:04:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:04:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:04:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-22T00:04:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:04:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:04:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:04:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd4 2024-09-22T00:04:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:04:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-22T00:04:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:04:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:04:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:04:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 2b8c112d-e876-476b-9041-ae5cbed1a24c 2024-09-22T00:04:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:04:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:04:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:04:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:04:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:04:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:04:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary5 2024-09-22T00:04:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-22T00:04:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:04:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary5 2024-09-22T00:04:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:04:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-22T00:04:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:04:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:04:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-22T00:04:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary6 2024-09-22T00:04:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary5 2024-09-22T00:04:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-22T00:04:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:04:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:04:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:04:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:04:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:04:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-22T00:04:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:04:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary5 2024-09-22T00:04:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:04:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:04:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:04:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-22T00:04:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-22T00:04:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:04:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:04:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-22T00:04:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:04:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-22T00:04:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:04:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-22T00:04:45.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:04:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 674 2024-09-22T00:04:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:04:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:04:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:04:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-22T00:04:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary6': 2024-09-22T00:04:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:04:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:04:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:04:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1a09aec897e3 2024-09-22T00:04:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1a09aec897e3 2024-09-22T00:04:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:04:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:04:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:04:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:04:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:04:44 2024 2024-09-22T00:04:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:04:44 2024 2024-09-22T00:04:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:04:44 2024 2024-09-22T00:04:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:04:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:04:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 8fb6c676-c05c-4963-b930-bfc650676205 2024-09-22T00:04:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:04:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:04:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:04:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-22T00:04:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd5 2024-09-22T00:04:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:04:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-22T00:04:45.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:04:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:04:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:04:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 78050739-5ac8-40ba-af8a-39a49d1abb92 2024-09-22T00:04:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:04:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:04:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:04:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:04:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:04:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:04:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary6 2024-09-22T00:04:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-22T00:04:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:04:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary6 2024-09-22T00:04:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:04:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-22T00:04:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:04:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:04:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:04:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:04:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-22T00:04:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary7 2024-09-22T00:04:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:04:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary6 2024-09-22T00:04:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-22T00:04:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:04:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary6 2024-09-22T00:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-22T00:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-22T00:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-22T00:04:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:04:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:04:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-22T00:04:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:04:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-22T00:04:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:04:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-22T00:04:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 673 2024-09-22T00:04:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:04:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:04:46.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:04:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:04:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 677 2024-09-22T00:04:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-22T00:04:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary7': 2024-09-22T00:04:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:04:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:04:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:04:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1a1bc80c7a02 2024-09-22T00:04:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1a1bc80c7a02 2024-09-22T00:04:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:04:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:04:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:04:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:04:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:04:45 2024 2024-09-22T00:04:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:04:45 2024 2024-09-22T00:04:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:04:45 2024 2024-09-22T00:04:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:04:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:04:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: a1d90309-6eda-42f2-99de-fbe088cec238 2024-09-22T00:04:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:04:46.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:04:46.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:04:46.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-22T00:04:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd6 2024-09-22T00:04:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:04:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-22T00:04:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:04:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:04:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:04:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 8f0b6dbb-f5fa-4d9a-8cf4-a202d5105f8b 2024-09-22T00:04:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:04:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:04:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:04:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:04:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:04:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:04:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary7 2024-09-22T00:04:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-22T00:04:46.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:04:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary7 2024-09-22T00:04:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:04:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-22T00:04:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:04:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:04:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:04:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:04:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:04:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary7 2024-09-22T00:04:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary7 2024-09-22T00:04:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-22T00:04:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-22T00:04:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary8 2024-09-22T00:04:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-22T00:04:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:04:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:04:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-22T00:04:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:04:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:04:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:04:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:04:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-22T00:04:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:04:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:04:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-22T00:04:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:04:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-22T00:04:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:04:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-22T00:04:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:04:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:04:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:04:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 676 2024-09-22T00:04:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:04:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-22T00:04:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 680 2024-09-22T00:04:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary8': 2024-09-22T00:04:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:04:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:04:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:04:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1a2dbff13bdb 2024-09-22T00:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1a2dbff13bdb 2024-09-22T00:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:04:46 2024 2024-09-22T00:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:04:46 2024 2024-09-22T00:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:04:46 2024 2024-09-22T00:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: d5d01b87-f76d-4510-bda7-91bc59eb57b2 2024-09-22T00:04:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:04:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:04:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:04:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-22T00:04:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd7 2024-09-22T00:04:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:04:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-22T00:04:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:04:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:04:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:04:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: d4642ec8-56bd-465b-9254-d8947286ccd1 2024-09-22T00:04:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:04:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:04:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:04:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:04:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:04:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:04:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary8 2024-09-22T00:04:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-22T00:04:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:04:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary8 2024-09-22T00:04:47.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:04:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-22T00:04:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:04:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:04:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-22T00:04:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary9 2024-09-22T00:04:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:04:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-22T00:04:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary8 2024-09-22T00:04:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:04:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:04:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:04:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:04:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-22T00:04:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary8 2024-09-22T00:04:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-22T00:04:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:04:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:04:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:04:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:04:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-22T00:04:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:04:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:04:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-22T00:04:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:04:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-22T00:04:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:04:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-22T00:04:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 679 2024-09-22T00:04:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 683 2024-09-22T00:04:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:04:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:04:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:04:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:04:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-22T00:04:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary9': 2024-09-22T00:04:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:04:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:04:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:04:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1a3ffe387451 2024-09-22T00:04:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1a3ffe387451 2024-09-22T00:04:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:04:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:04:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:04:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:04:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:04:47 2024 2024-09-22T00:04:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:04:47 2024 2024-09-22T00:04:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:04:47 2024 2024-09-22T00:04:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:04:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:04:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: a21c9573-e1b1-4890-8240-0bb25f6afb4e 2024-09-22T00:04:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:04:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:04:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:04:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-22T00:04:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd8 2024-09-22T00:04:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:04:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-22T00:04:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:04:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:04:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:04:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: cae70d6f-b85f-49a6-a1db-ac7bf4fa4027 2024-09-22T00:04:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:04:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:04:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:04:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:04:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:04:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:04:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary9 2024-09-22T00:04:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-22T00:04:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:04:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary9 2024-09-22T00:04:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:04:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-22T00:04:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:04:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:04:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary9 2024-09-22T00:04:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-22T00:04:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary10 2024-09-22T00:04:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-22T00:04:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:04:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary9 2024-09-22T00:04:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:04:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-22T00:04:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-22T00:04:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:04:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:04:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:04:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:04:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:04:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:04:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:04:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-22T00:04:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:04:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:04:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:04:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:04:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:04:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-22T00:04:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:04:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:04:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-22T00:04:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:04:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-22T00:04:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 682 2024-09-22T00:04:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 686 2024-09-22T00:04:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:04:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-22T00:04:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:04:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:04:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:04:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary10': 2024-09-22T00:04:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:04:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:04:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:04:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1a4dd0b71307 2024-09-22T00:04:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1a4dd0b71307 2024-09-22T00:04:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:04:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:04:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:04:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:04:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:04:48 2024 2024-09-22T00:04:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:04:48 2024 2024-09-22T00:04:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:04:48 2024 2024-09-22T00:04:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:04:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:04:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 7d09dfcc-3937-45be-b0b7-d2da1ba52854 2024-09-22T00:04:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:04:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:04:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:04:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-22T00:04:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:04:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:04:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:04:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd9 2024-09-22T00:04:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:04:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-22T00:04:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:04:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:04:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:04:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 59d53e42-800a-486d-ab76-60dcc1e2072b 2024-09-22T00:04:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:04:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:04:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:04:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:04:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:04:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:04:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:04:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary10 2024-09-22T00:04:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-22T00:04:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:04:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary10 2024-09-22T00:04:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:04:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-22T00:04:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:04:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:04:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:04:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:04:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:04:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:04:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 53431 2024-09-22T00:04:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary10 2024-09-22T00:04:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary10 2024-09-22T00:04:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-22T00:04:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:04:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-22T00:04:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 685 2024-09-22T00:04:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:04:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:04:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:04:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 690 2024-09-22T00:04:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:04:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:04:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:04:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 688 2024-09-22T00:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:04:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:04:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 689 2024-09-22T00:04:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:04:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:04:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:04:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:51.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:51.094 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:04:51.095+0000 7fab5fe19640 -1 reset not still connected to 0x562416c29ad0 2024-09-22T00:04:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:04:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:04:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:04:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:04:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:04:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:04:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 692 2024-09-22T00:04:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 691 2024-09-22T00:04:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 693 2024-09-22T00:04:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 694 2024-09-22T00:04:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:04:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:04:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:04:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:04:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:04:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:04:53.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 695 2024-09-22T00:04:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 696 2024-09-22T00:04:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:04:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:04:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:04:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:04:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:04:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:04:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 697 2024-09-22T00:04:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:04:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:04:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:04:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:04:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:04:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:04:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 698 2024-09-22T00:04:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:04:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:04:56.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:04:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 700 2024-09-22T00:04:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 699 2024-09-22T00:04:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:56.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:04:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:04:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:04:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:04:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:04:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:04:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:04:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:04:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:04:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 702 2024-09-22T00:04:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 701 2024-09-22T00:04:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:57.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:04:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:04:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:04:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 704 2024-09-22T00:04:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 703 2024-09-22T00:04:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:04:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:04:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:04:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 705 2024-09-22T00:04:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 706 2024-09-22T00:04:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:04:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:04:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:04:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:04:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:04:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:04:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:04:59.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:04:59.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:04:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 708 2024-09-22T00:04:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 707 2024-09-22T00:04:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:04:59.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:05:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:05:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:05:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:05:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:05:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:05:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 709 2024-09-22T00:05:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 710 2024-09-22T00:05:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:05:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:05:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:05:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:05:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:05:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:05:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 711 2024-09-22T00:05:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 712 2024-09-22T00:05:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:05:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:05:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:05:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:05:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:05:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:05:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 714 2024-09-22T00:05:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 713 2024-09-22T00:05:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:05:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:05:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:05:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:05:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:05:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:05:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 715 2024-09-22T00:05:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 716 2024-09-22T00:05:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:03.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:05:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:05:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:05:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:05:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:05:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:05:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 718 2024-09-22T00:05:04.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 717 2024-09-22T00:05:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:05:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:05:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:05:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:05:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:05:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:05:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:04.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 719 2024-09-22T00:05:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 720 2024-09-22T00:05:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:05:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:05:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:05:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:05:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:05:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:05:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:05:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:05:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:05:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:05:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:05:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:05:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 722 2024-09-22T00:05:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 721 2024-09-22T00:05:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 723 2024-09-22T00:05:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 724 2024-09-22T00:05:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:05:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:05:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:05:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:05:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:05:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:05:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 725 2024-09-22T00:05:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 726 2024-09-22T00:05:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:05:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:05:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:05:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:05:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:05:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:05:09.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 727 2024-09-22T00:05:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 728 2024-09-22T00:05:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:05:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:05:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:05:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:05:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:05:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:05:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 729 2024-09-22T00:05:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 730 2024-09-22T00:05:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:05:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:05:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:05:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:05:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:05:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:05:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:11.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 732 2024-09-22T00:05:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:05:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:05:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:05:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:05:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:05:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:05:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 731 2024-09-22T00:05:11.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 733 2024-09-22T00:05:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 734 2024-09-22T00:05:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:05:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:05:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:05:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:05:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:05:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:05:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:13.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 735 2024-09-22T00:05:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 736 2024-09-22T00:05:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:05:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:05:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:05:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:05:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:05:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:05:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 737 2024-09-22T00:05:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 738 2024-09-22T00:05:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:05:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:05:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:05:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:05:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:05:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:05:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 740 2024-09-22T00:05:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 739 2024-09-22T00:05:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:05:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:05:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:05:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:05:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:16.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:05:16.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:05:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:05:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:05:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:05:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:05:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:05:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:05:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 742 2024-09-22T00:05:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 741 2024-09-22T00:05:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 744 2024-09-22T00:05:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 743 2024-09-22T00:05:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:05:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:05:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:05:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:05:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:05:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:05:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:18.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 745 2024-09-22T00:05:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 746 2024-09-22T00:05:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:05:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:05:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:05:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:05:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:05:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:05:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 747 2024-09-22T00:05:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 748 2024-09-22T00:05:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:05:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:05:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:05:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:05:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:05:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:05:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 749 2024-09-22T00:05:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 750 2024-09-22T00:05:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:05:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:05:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:05:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:05:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:05:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:05:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:21.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:05:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:05:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:05:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:05:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:05:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:05:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 752 2024-09-22T00:05:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 751 2024-09-22T00:05:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:23.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 754 2024-09-22T00:05:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:23.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:23.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:05:23.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:23.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:23.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:05:23.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:05:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:05:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:05:23.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:05:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 753 2024-09-22T00:05:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 755 2024-09-22T00:05:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 756 2024-09-22T00:05:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:05:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:05:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:05:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:05:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:05:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:05:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 758 2024-09-22T00:05:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 757 2024-09-22T00:05:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:05:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:05:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:05:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:05:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:05:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:05:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 760 2024-09-22T00:05:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:05:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:05:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:05:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:26.093 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:05:26.094+0000 7fab5fe19640 -1 reset not still connected to 0x5623fe71bba0 2024-09-22T00:05:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:05:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:05:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:05:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:05:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:05:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:05:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:05:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:05:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:05:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 761 2024-09-22T00:05:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 759 2024-09-22T00:05:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 762 2024-09-22T00:05:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:05:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:05:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:05:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:05:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:05:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:05:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 763 2024-09-22T00:05:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 764 2024-09-22T00:05:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 766 2024-09-22T00:05:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 765 2024-09-22T00:05:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:05:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:05:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:05:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 768 2024-09-22T00:05:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 767 2024-09-22T00:05:29.919 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:05:29.918+0000 7f54f2d95640 -1 reset not still connected to 0x5617f563d520 2024-09-22T00:05:29.919 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:05:29.918+0000 7f54f2d95640 -1 reset not still connected to 0x5617fbb91ba0 2024-09-22T00:05:29.919 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:05:29.918+0000 7f54f2d95640 -1 reset not still connected to 0x5617fd635c70 2024-09-22T00:05:29.920 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:05:29.918+0000 7f54f2d95640 -1 reset not still connected to 0x561805229a00 2024-09-22T00:05:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:05:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:05:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:05:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:05:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:05:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:05:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:05:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:05:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:05:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 769 2024-09-22T00:05:30.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 772 2024-09-22T00:05:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:05:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:05:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:05:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:05:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:05:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:05:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 771 2024-09-22T00:05:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 770 2024-09-22T00:05:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:05:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:05:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:05:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:05:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:05:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:05:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 773 2024-09-22T00:05:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:05:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:05:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:05:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 774 2024-09-22T00:05:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:05:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:05:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:05:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 776 2024-09-22T00:05:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 775 2024-09-22T00:05:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:05:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:05:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:05:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:05:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:05:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:05:34.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 778 2024-09-22T00:05:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 777 2024-09-22T00:05:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:05:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:05:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:05:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:05:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:05:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:05:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:05:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:05:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:05:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 780 2024-09-22T00:05:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 779 2024-09-22T00:05:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:05:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:05:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:05:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:05:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:05:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:05:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 782 2024-09-22T00:05:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 781 2024-09-22T00:05:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:05:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:05:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:05:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 783 2024-09-22T00:05:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 784 2024-09-22T00:05:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:05:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:05:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:05:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:05:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:05:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:05:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 785 2024-09-22T00:05:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 786 2024-09-22T00:05:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:05:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:05:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:05:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:05:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:05:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:05:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 788 2024-09-22T00:05:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 787 2024-09-22T00:05:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:05:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:05:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:05:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:05:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:05:40.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:05:40.020 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:05:40.014+0000 7f54f2d95640 -1 reset not still connected to 0x5617fd634c30 2024-09-22T00:05:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 790 2024-09-22T00:05:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 789 2024-09-22T00:05:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:05:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:05:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:05:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:05:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:05:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:05:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:05:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:05:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:05:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:05:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:05:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:05:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 791 2024-09-22T00:05:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 792 2024-09-22T00:05:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 793 2024-09-22T00:05:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 794 2024-09-22T00:05:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:05:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:05:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:05:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:05:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:05:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:05:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 795 2024-09-22T00:05:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 796 2024-09-22T00:05:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:05:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:05:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:05:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:05:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:05:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:05:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 798 2024-09-22T00:05:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 797 2024-09-22T00:05:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:05:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:05:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:05:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:05:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:05:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:05:45.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 800 2024-09-22T00:05:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 799 2024-09-22T00:05:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:05:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:05:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:05:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:05:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:05:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:05:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:05:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:05:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:05:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:05:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:05:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:05:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 802 2024-09-22T00:05:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 801 2024-09-22T00:05:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 804 2024-09-22T00:05:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 803 2024-09-22T00:05:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:05:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:05:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:05:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:05:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 805 2024-09-22T00:05:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 806 2024-09-22T00:05:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:49.014 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:05:49.007+0000 7fab1f496640 -1 reset not still connected to 0x55fa5eb07450 2024-09-22T00:05:49.014 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:05:49.007+0000 7fab1f496640 -1 reset not still connected to 0x55fa5fcfbd40 2024-09-22T00:05:49.015 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:05:49.011+0000 7fab1f496640 -1 reset not still connected to 0x55fa60a0e0d0 2024-09-22T00:05:49.015 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:05:49.011+0000 7fab1f496640 -1 reset not still connected to 0x55fa61b85520 2024-09-22T00:05:49.015 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:05:49.011+0000 7fab1f496640 -1 reset not still connected to 0x55fa7610b110 2024-09-22T00:05:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:05:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:05:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:05:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:05:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:05:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:05:50.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 808 2024-09-22T00:05:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 807 2024-09-22T00:05:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:05:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:05:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:05:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:05:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:05:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:05:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 809 2024-09-22T00:05:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 810 2024-09-22T00:05:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:05:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:05:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:05:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:05:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:05:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:05:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:05:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:05:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:05:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:05:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:05:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:05:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 812 2024-09-22T00:05:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 811 2024-09-22T00:05:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:52.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 813 2024-09-22T00:05:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 814 2024-09-22T00:05:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:05:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:05:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:05:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:05:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:05:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:05:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 816 2024-09-22T00:05:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 815 2024-09-22T00:05:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:54.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:54.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:05:54.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:54.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:54.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:05:54.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:05:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:05:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:05:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:05:54.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 817 2024-09-22T00:05:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 818 2024-09-22T00:05:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:05:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:55.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:05:55.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:05:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:05:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:05:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:05:55.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 819 2024-09-22T00:05:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 820 2024-09-22T00:05:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:05:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:05:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:05:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:05:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:05:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:05:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 821 2024-09-22T00:05:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 822 2024-09-22T00:05:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:05:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:05:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:05:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:05:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:05:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:05:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 823 2024-09-22T00:05:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 824 2024-09-22T00:05:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:05:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:05:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:05:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:05:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:05:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:05:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 825 2024-09-22T00:05:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 826 2024-09-22T00:05:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:05:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:05:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:05:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:05:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:05:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:05:59.848 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:05:59.843+0000 7f84953c5640 -1 reset not still connected to 0x5636f85f0d00 2024-09-22T00:05:59.851 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:05:59.847+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b32ea0 2024-09-22T00:05:59.851 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:05:59.847+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b332b0 2024-09-22T00:05:59.859 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:05:59.851+0000 7f84953c5640 -1 reset not still connected to 0x5636f8ec80d0 2024-09-22T00:05:59.859 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:05:59.851+0000 7f84953c5640 -1 reset not still connected to 0x5636f8f3ec30 2024-09-22T00:05:59.859 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:05:59.851+0000 7f84953c5640 -1 reset not still connected to 0x5636f9c289c0 2024-09-22T00:05:59.860 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:05:59.855+0000 7f84953c5640 -1 reset not still connected to 0x5636fdf66ea0 2024-09-22T00:05:59.860 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:05:59.855+0000 7f84953c5640 -1 reset not still connected to 0x56370b3f5930 2024-09-22T00:05:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 827 2024-09-22T00:05:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 828 2024-09-22T00:05:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:05:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:05:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:05:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:05:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:05:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:05:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:05:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:05:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:05:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:05:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 830 2024-09-22T00:06:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 829 2024-09-22T00:06:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:06:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:06:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:06:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:06:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:06:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:06:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 831 2024-09-22T00:06:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 832 2024-09-22T00:06:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:06:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:06:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:06:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:06:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:06:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:06:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 834 2024-09-22T00:06:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:06:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:06:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:06:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:06:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:06:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:06:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 833 2024-09-22T00:06:02.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 836 2024-09-22T00:06:03.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 835 2024-09-22T00:06:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:06:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:06:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:06:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:06:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:06:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:06:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 838 2024-09-22T00:06:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:06:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:06:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:06:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 837 2024-09-22T00:06:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:06:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:06:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:06:04.998 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:06:04.990+0000 7f54f2d95640 -1 reset not still connected to 0x5617f7a3e820 2024-09-22T00:06:04.999 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:06:04.994+0000 7f54f2d95640 -1 reset not still connected to 0x5617f86364e0 2024-09-22T00:06:05.000 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:06:04.998+0000 7f54f2d95640 -1 reset not still connected to 0x5617fd634c30 2024-09-22T00:06:05.000 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:06:04.998+0000 7f54f2d95640 -1 reset not still connected to 0x56180480ad00 2024-09-22T00:06:05.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:05.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 839 2024-09-22T00:06:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 840 2024-09-22T00:06:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:06:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:06:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:06:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:06:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:06:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:06:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:06:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:06:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:06:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:06:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:06:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:06:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 842 2024-09-22T00:06:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 841 2024-09-22T00:06:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 844 2024-09-22T00:06:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:06:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:06:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:06:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:06:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:06:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:06:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 843 2024-09-22T00:06:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 845 2024-09-22T00:06:08.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 846 2024-09-22T00:06:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:06:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:06:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:06:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:06:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:06:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:06:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 847 2024-09-22T00:06:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 848 2024-09-22T00:06:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:06:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:06:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:06:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:06:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:06:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:06:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 849 2024-09-22T00:06:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 850 2024-09-22T00:06:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:06:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:06:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:06:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:06:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:06:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:06:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:06:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:06:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:06:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:06:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:06:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:06:12.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 852 2024-09-22T00:06:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 851 2024-09-22T00:06:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 853 2024-09-22T00:06:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:06:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:06:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:06:13.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:06:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:06:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:06:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 854 2024-09-22T00:06:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 856 2024-09-22T00:06:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 855 2024-09-22T00:06:13.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:06:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:06:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:06:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:06:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:06:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:06:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 858 2024-09-22T00:06:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 857 2024-09-22T00:06:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:15.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:06:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:06:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:06:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:06:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:06:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:06:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 860 2024-09-22T00:06:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:06:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:06:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:06:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 859 2024-09-22T00:06:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:06:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:06:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:06:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:06:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:06:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:06:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:06:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:06:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:06:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 862 2024-09-22T00:06:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 861 2024-09-22T00:06:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 863 2024-09-22T00:06:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:06:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:06:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:06:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 864 2024-09-22T00:06:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:06:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:06:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:06:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 866 2024-09-22T00:06:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 865 2024-09-22T00:06:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:06:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:06:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:06:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:06:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:06:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:06:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 868 2024-09-22T00:06:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 867 2024-09-22T00:06:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:06:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:06:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:06:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:06:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:06:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:06:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 869 2024-09-22T00:06:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 870 2024-09-22T00:06:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:06:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:06:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:06:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:06:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:06:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:06:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 872 2024-09-22T00:06:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 871 2024-09-22T00:06:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:06:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:06:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:06:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:06:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:06:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:06:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 873 2024-09-22T00:06:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 874 2024-09-22T00:06:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:06:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:06:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:06:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:06:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:06:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:06:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 875 2024-09-22T00:06:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 876 2024-09-22T00:06:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:06:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:06:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:06:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:06:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:06:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:06:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:24.074 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:06:24.073+0000 7f54f2d95640 -1 reset not still connected to 0x561805229a00 2024-09-22T00:06:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 877 2024-09-22T00:06:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 878 2024-09-22T00:06:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:06:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:06:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:06:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:06:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:06:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:06:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 879 2024-09-22T00:06:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 880 2024-09-22T00:06:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:06:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:06:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:06:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:06:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:06:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:06:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 881 2024-09-22T00:06:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 882 2024-09-22T00:06:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:06:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:06:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:06:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:06:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:06:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:06:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 884 2024-09-22T00:06:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:06:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:06:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:06:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:06:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:06:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:06:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 883 2024-09-22T00:06:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 885 2024-09-22T00:06:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 886 2024-09-22T00:06:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:06:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:06:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:06:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:06:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:06:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:06:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 887 2024-09-22T00:06:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 888 2024-09-22T00:06:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:06:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:06:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:06:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:06:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:06:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:06:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:06:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:06:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:06:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 890 2024-09-22T00:06:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 889 2024-09-22T00:06:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:06:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:06:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:06:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 892 2024-09-22T00:06:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:06:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:06:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:06:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:06:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:06:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:06:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 891 2024-09-22T00:06:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 893 2024-09-22T00:06:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:06:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:06:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:06:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:06:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:06:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:06:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 894 2024-09-22T00:06:33.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 896 2024-09-22T00:06:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 895 2024-09-22T00:06:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:06:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:06:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:06:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:06:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:06:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:06:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 898 2024-09-22T00:06:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 897 2024-09-22T00:06:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:06:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:06:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:06:35.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:35.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:06:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:06:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:06:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 899 2024-09-22T00:06:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 900 2024-09-22T00:06:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:06:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:06:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:06:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:06:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:06:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:06:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:06:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:06:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:06:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:06:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:06:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:06:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 902 2024-09-22T00:06:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 901 2024-09-22T00:06:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 903 2024-09-22T00:06:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 904 2024-09-22T00:06:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:06:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:06:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:06:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:06:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:06:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:06:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 905 2024-09-22T00:06:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 906 2024-09-22T00:06:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:06:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:06:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:06:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:06:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:06:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:06:39.876 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:06:39.871+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a3640f29c0 2024-09-22T00:06:39.876 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:06:39.871+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a367cd8680 2024-09-22T00:06:39.876 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:06:39.871+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a36b26cea0 2024-09-22T00:06:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:06:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:06:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:06:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:06:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:06:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:06:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:06:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:06:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:06:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 907 2024-09-22T00:06:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:06:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:06:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:06:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 908 2024-09-22T00:06:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 910 2024-09-22T00:06:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:06:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:06:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:06:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:06:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:06:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:06:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 909 2024-09-22T00:06:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 912 2024-09-22T00:06:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 911 2024-09-22T00:06:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 914 2024-09-22T00:06:43.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 913 2024-09-22T00:06:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 916 2024-09-22T00:06:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 915 2024-09-22T00:06:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:06:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:06:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:06:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:06:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:06:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:06:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:06:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:06:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:06:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 918 2024-09-22T00:06:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:06:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:06:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:06:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:06:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:06:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:06:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 917 2024-09-22T00:06:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:06:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:06:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:06:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:06:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:06:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:06:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:06:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:06:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:06:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 919 2024-09-22T00:06:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 921 2024-09-22T00:06:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:06:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:06:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:06:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:06:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:06:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:06:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 920 2024-09-22T00:06:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 922 2024-09-22T00:06:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:48.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 923 2024-09-22T00:06:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 924 2024-09-22T00:06:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:06:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:06:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:06:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:06:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:06:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:06:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 926 2024-09-22T00:06:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 925 2024-09-22T00:06:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:06:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:06:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:06:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:06:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:06:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:06:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 927 2024-09-22T00:06:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 928 2024-09-22T00:06:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:06:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:06:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:06:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:06:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:06:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:06:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 930 2024-09-22T00:06:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 929 2024-09-22T00:06:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:51.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:51.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:06:51.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:51.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:06:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:06:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:06:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:06:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:06:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 931 2024-09-22T00:06:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 932 2024-09-22T00:06:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:06:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:06:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:06:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:06:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:06:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:06:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 933 2024-09-22T00:06:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 934 2024-09-22T00:06:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:06:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:06:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:06:52.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:52.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:52.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:06:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:06:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:06:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 935 2024-09-22T00:06:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 936 2024-09-22T00:06:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:06:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:06:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:06:53.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:06:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:06:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:06:53.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 938 2024-09-22T00:06:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 937 2024-09-22T00:06:54.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:54.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:06:54.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:54.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:54.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:06:54.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:06:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:06:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:06:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:06:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:06:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:06:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:06:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:06:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:06:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:06:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 939 2024-09-22T00:06:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 940 2024-09-22T00:06:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:06:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:06:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:06:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 941 2024-09-22T00:06:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 942 2024-09-22T00:06:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:06:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:06:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:06:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 943 2024-09-22T00:06:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 944 2024-09-22T00:06:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:06:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:06:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:06:58.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:58.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:06:58.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:58.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:58.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:06:58.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:06:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 946 2024-09-22T00:06:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 945 2024-09-22T00:06:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:06:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:06:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:06:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:06:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:06:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:06:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:06:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:06:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:06:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:06:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 948 2024-09-22T00:07:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 947 2024-09-22T00:07:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:07:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:07:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:07:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:07:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:07:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:07:00.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:01.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 949 2024-09-22T00:07:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 950 2024-09-22T00:07:01.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:01.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:07:01.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:01.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:01.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:07:01.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:07:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:07:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:07:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:07:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:07:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:07:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:07:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:07:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:07:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:07:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 952 2024-09-22T00:07:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 951 2024-09-22T00:07:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:02.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 954 2024-09-22T00:07:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:07:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:07:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:07:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:07:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:07:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:07:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 953 2024-09-22T00:07:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 955 2024-09-22T00:07:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 956 2024-09-22T00:07:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:07:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:07:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:07:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:07:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:07:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:07:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 958 2024-09-22T00:07:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 957 2024-09-22T00:07:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:07:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:07:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:07:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:07:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:07:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:07:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 960 2024-09-22T00:07:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 959 2024-09-22T00:07:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:07:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:07:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:07:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:07:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:07:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:07:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:07:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:07:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:07:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:07:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:07:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:07:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 961 2024-09-22T00:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 962 2024-09-22T00:07:07.077 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:07:07.080+0000 7fab5fe19640 -1 reset not still connected to 0x562416c29ad0 2024-09-22T00:07:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 963 2024-09-22T00:07:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 964 2024-09-22T00:07:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:07:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:07:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:07:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:07:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:07:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:07:08.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 966 2024-09-22T00:07:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 965 2024-09-22T00:07:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:07:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:07:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:07:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:07:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 53553 2024-09-22T00:07:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 967 2024-09-22T00:07:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 968 2024-09-22T00:07:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:07:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 53685 2024-09-22T00:07:10.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:10.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:07:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:07:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:07:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 969 2024-09-22T00:07:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:07:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:07:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:07:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:07:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 53812 2024-09-22T00:07:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:07:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 53959 2024-09-22T00:07:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:07:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:07:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:07:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 970 2024-09-22T00:07:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 971 2024-09-22T00:07:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:07:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 54106 2024-09-22T00:07:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:07:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:07:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:07:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:07:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:07:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:07:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 972 2024-09-22T00:07:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:07:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 54254 2024-09-22T00:07:14.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 973 2024-09-22T00:07:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:07:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:07:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 54402 2024-09-22T00:07:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:07:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 54550 2024-09-22T00:07:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:07:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 54718 2024-09-22T00:07:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:07:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:07:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 53432 2024-09-22T00:09:37.153 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:09:37.149+0000 7fab5fe19640 -1 reset not still connected to 0x5623fd06ab60 2024-09-22T00:09:37.153 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:09:37.149+0000 7fab5fe19640 -1 reset not still connected to 0x5623fe71bba0 2024-09-22T00:09:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:09:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:09:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:09:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 53554 2024-09-22T00:09:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:09:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:09:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:09:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 53686 2024-09-22T00:09:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:09:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:09:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:09:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 53813 2024-09-22T00:09:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:09:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:09:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:09:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 53960 2024-09-22T00:09:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:09:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:09:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:09:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 54107 2024-09-22T00:09:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:09:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:09:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:09:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 54255 2024-09-22T00:09:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:09:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:09:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:09:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 54403 2024-09-22T00:09:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:09:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:09:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:09:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 54551 2024-09-22T00:09:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:09:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:09:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:09:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 54719 2024-09-22T00:09:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:09:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:09:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:09:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:09:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:09:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 1 2024-09-22T00:09:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:09:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd0 2024-09-22T00:09:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary1 2024-09-22T00:09:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary1 2024-09-22T00:09:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:09:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary1 2024-09-22T00:09:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:09:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:09:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 2 2024-09-22T00:09:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd1 2024-09-22T00:09:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary2 2024-09-22T00:09:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary2 2024-09-22T00:09:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:09:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary2 2024-09-22T00:09:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:09:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:09:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 3 2024-09-22T00:09:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd2 2024-09-22T00:09:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary3 2024-09-22T00:09:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary3 2024-09-22T00:09:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:09:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:09:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary3 2024-09-22T00:09:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:09:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 4 2024-09-22T00:09:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd3 2024-09-22T00:09:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:09:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary4 2024-09-22T00:09:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary4 2024-09-22T00:09:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 5 2024-09-22T00:09:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:09:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:09:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary4 2024-09-22T00:09:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd4 2024-09-22T00:09:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary5 2024-09-22T00:09:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary5 2024-09-22T00:09:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:09:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary5 2024-09-22T00:09:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:09:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:09:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 6 2024-09-22T00:09:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd5 2024-09-22T00:09:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:09:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:09:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 7 2024-09-22T00:09:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary6 2024-09-22T00:09:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary6 2024-09-22T00:09:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:09:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary6 2024-09-22T00:09:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:09:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:09:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd6 2024-09-22T00:09:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary7 2024-09-22T00:09:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary7 2024-09-22T00:09:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:09:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary7 2024-09-22T00:09:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:09:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:09:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:09:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:09:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 60874 2024-09-22T00:09:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 10 2024-09-22T00:09:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd9 2024-09-22T00:09:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary10 2024-09-22T00:09:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary10 2024-09-22T00:09:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:09:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary10 2024-09-22T00:09:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 9 2024-09-22T00:09:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd8 2024-09-22T00:09:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary9 2024-09-22T00:09:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary9 2024-09-22T00:09:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:09:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary9 2024-09-22T00:09:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 8 2024-09-22T00:09:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd7 2024-09-22T00:09:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary8 2024-09-22T00:09:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary8 2024-09-22T00:09:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:09:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary8 2024-09-22T00:09:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:09:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-22T00:09:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-22T00:09:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:09:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-22T00:09:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:09:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-22T00:09:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:09:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-22T00:09:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:09:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-22T00:09:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:09:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-22T00:09:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:09:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-22T00:09:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:09:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-22T00:09:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:09:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-22T00:09:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:10:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=8b73c40b9791bdf7ee60d99993325d95 2024-09-22T00:10:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-22T00:10:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=28f395986814b4292f0971e1b301a109 2024-09-22T00:10:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-22T00:10:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-22T00:10:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:10:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-22T00:10:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-22T00:10:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:10:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-22T00:10:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=fc1e139239a8dc8e2cca81fb1b132b9f 2024-09-22T00:10:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-22T00:10:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=93808609080059515c367279f8edacac 2024-09-22T00:10:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-22T00:10:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=c5e431ea57f6b398b4d07fbf3231b363 2024-09-22T00:10:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-22T00:10:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=bb8a4d4ab16fa7465fd09204aeb12064 2024-09-22T00:10:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-22T00:10:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-22T00:10:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:10:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-22T00:10:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=3607fb0bf757e3b27bab00c3f777e0ee 2024-09-22T00:10:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-22T00:10:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-22T00:10:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:10:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-22T00:10:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=eca056fbd0ece1703df150eedccfbb6f 2024-09-22T00:10:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-22T00:10:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-22T00:10:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:10:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-22T00:10:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-22T00:10:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:10:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-22T00:10:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-22T00:10:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:10:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-22T00:10:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=e35e5ac5d82da35687f456b8c44c3c2c 2024-09-22T00:10:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-22T00:10:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-22T00:10:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:10:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-22T00:10:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=7a8dbb279ae969377e30805ab10c4116 2024-09-22T00:10:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-22T00:10:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-22T00:10:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:10:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-22T00:10:23.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-22T00:10:23.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:23.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:23.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:10:23.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-22T00:10:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:10:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:10:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-22T00:10:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:10:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-22T00:10:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:10:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:10:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:10:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:10:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:10:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-22T00:10:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:10:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:10:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:10:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-22T00:10:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:10:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-22T00:10:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:10:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:10:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:10:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:10:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-22T00:10:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:10:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:10:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-22T00:10:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:10:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-22T00:10:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:10:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-22T00:10:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:10:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-22T00:10:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:10:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-22T00:10:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:10:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-22T00:10:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:10:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:10:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 859a2725-431b-4842-85c5-dceb1043514d 2024-09-22T00:10:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:10:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:10:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:20 2024-09-22T00:10:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:10:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:10:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:24' 2024-09-22T00:10:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:10:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:10:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:10:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: a1d90309-6eda-42f2-99de-fbe088cec238 2024-09-22T00:10:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:10:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:10:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:20 2024-09-22T00:10:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:10:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:10:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25' 2024-09-22T00:10:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:10:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:10:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-22T00:10:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:10:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:10:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-22T00:10:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:10:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:26.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:10:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-22T00:10:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:10:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:10:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-22T00:10:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:10:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:10:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-22T00:10:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:10:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:10:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-22T00:10:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:10:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:10:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-22T00:10:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:10:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:10:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:10:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: d5d01b87-f76d-4510-bda7-91bc59eb57b2 2024-09-22T00:10:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25 2024-09-22T00:10:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:10:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:10:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2820096.0,"last_snapshot_bytes":5476352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963632,"remote_snapshot_timestamp":1726963632,"replay_state":"idle"} 2024-09-22T00:10:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:05' 2024-09-22T00:10:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:10:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-22T00:10:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:10:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:10:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-22 00:10:25' 2024-09-22T00:10:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:10:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:25' 2024-09-22T00:10:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:10:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 52725bae-bfe7-4465-ad3b-965fe6fc62de 2024-09-22T00:10:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25 2024-09-22T00:10:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:10:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:10:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25' 2024-09-22T00:10:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:10:26.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:26.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:26.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:26.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-22 00:10:25' 2024-09-22T00:10:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:10:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:25' 2024-09-22T00:10:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:10:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 183dbf3b-282a-4c6d-9ed1-e9865ba1f6db 2024-09-22T00:10:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25 2024-09-22T00:10:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:10:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:10:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3309568.0,"last_snapshot_bytes":6455296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963625,"remote_snapshot_timestamp":1726963625,"replay_state":"idle"} 2024-09-22T00:10:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:05' 2024-09-22T00:10:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:10:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:10:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: a21c9573-e1b1-4890-8240-0bb25f6afb4e 2024-09-22T00:10:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25 2024-09-22T00:10:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:10:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:10:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:26' 2024-09-22T00:10:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:26.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:26.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:26.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:10:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-22 00:10:25' 2024-09-22T00:10:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:10:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:25' 2024-09-22T00:10:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:10:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8fb6c676-c05c-4963-b930-bfc650676205 2024-09-22T00:10:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25 2024-09-22T00:10:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:10:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:10:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:26' 2024-09-22T00:10:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:10:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 7d09dfcc-3937-45be-b0b7-d2da1ba52854 2024-09-22T00:10:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25 2024-09-22T00:10:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:10:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:10:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:26' 2024-09-22T00:10:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:10:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:10:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-22 00:10:25' 2024-09-22T00:10:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:10:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:25' 2024-09-22T00:10:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-22 00:10:25' 2024-09-22T00:10:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:10:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:25' 2024-09-22T00:10:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-22 00:10:25' 2024-09-22T00:10:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:10:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:25' 2024-09-22T00:10:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:10:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: ffe30d9d-5e74-4ba9-81a4-1f3439aea154 2024-09-22T00:10:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25 2024-09-22T00:10:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:10:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:10:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:26' 2024-09-22T00:10:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:10:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 7d3beb22-c450-46bd-9c24-aa8184e646ef 2024-09-22T00:10:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25 2024-09-22T00:10:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:10:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:10:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1040384.0,"last_snapshot_bytes":1937408,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963627,"remote_snapshot_timestamp":1726963627,"replay_state":"idle"} 2024-09-22T00:10:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:05' 2024-09-22T00:10:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:10:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-22 00:10:25' 2024-09-22T00:10:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:10:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:25' 2024-09-22T00:10:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:10:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-22 00:10:25' 2024-09-22T00:10:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:10:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:25' 2024-09-22T00:10:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3309568.0,"last_snapshot_bytes":6455296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963625,"remote_snapshot_timestamp":1726963625,"replay_state":"idle"}2024-09-22 00:10:05image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1040384.0,"last_snapshot_bytes":1937408,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963627,"remote_snapshot_timestamp":1726963627,"replay_state":"idle"}2024-09-22 00:10:05image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2820096.0,"last_snapshot_bytes":5476352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963632,"remote_snapshot_timestamp":1726963632,"replay_state":"idle"}2024-09-22 00:10:05image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26' 2024-09-22T00:10:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:10:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3309568.0,"last_snapshot_bytes":6455296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963625,"remote_snapshot_timestamp":1726963625,"replay_state":"idle"}2024-09-22 00:10:05image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1040384.0,"last_snapshot_bytes":1937408,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963627,"remote_snapshot_timestamp":1726963627,"replay_state":"idle"}2024-09-22 00:10:05image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2820096.0,"last_snapshot_bytes":5476352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963632,"remote_snapshot_timestamp":1726963632,"replay_state":"idle"}2024-09-22 00:10:05image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26' 2024-09-22T00:10:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:10:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:10:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:10:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-22T00:10:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:10:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-22T00:10:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:26.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:10:26.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:26.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:26.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:26.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3309568.0,"last_snapshot_bytes":6455296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963625,"remote_snapshot_timestamp":1726963625,"replay_state":"idle"}2024-09-22 00:10:05image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1040384.0,"last_snapshot_bytes":1937408,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963627,"remote_snapshot_timestamp":1726963627,"replay_state":"idle"}2024-09-22 00:10:05image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2820096.0,"last_snapshot_bytes":5476352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963632,"remote_snapshot_timestamp":1726963632,"replay_state":"idle"}2024-09-22 00:10:05image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26' 2024-09-22T00:10:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:10:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:10:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3309568.0,"last_snapshot_bytes":6455296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963625,"remote_snapshot_timestamp":1726963625,"replay_state":"idle"}2024-09-22 00:10:05image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1040384.0,"last_snapshot_bytes":1937408,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963627,"remote_snapshot_timestamp":1726963627,"replay_state":"idle"}2024-09-22 00:10:05image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2820096.0,"last_snapshot_bytes":5476352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963632,"remote_snapshot_timestamp":1726963632,"replay_state":"idle"}2024-09-22 00:10:05image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26' 2024-09-22T00:10:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:10:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-22T00:10:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:10:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:10:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3309568.0,"last_snapshot_bytes":6455296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963625,"remote_snapshot_timestamp":1726963625,"replay_state":"idle"}2024-09-22 00:10:05image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1040384.0,"last_snapshot_bytes":1937408,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963627,"remote_snapshot_timestamp":1726963627,"replay_state":"idle"}2024-09-22 00:10:05image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2820096.0,"last_snapshot_bytes":5476352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963632,"remote_snapshot_timestamp":1726963632,"replay_state":"idle"}2024-09-22 00:10:05image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26' 2024-09-22T00:10:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:10:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:10:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3309568.0,"last_snapshot_bytes":6455296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963625,"remote_snapshot_timestamp":1726963625,"replay_state":"idle"}2024-09-22 00:10:05image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1040384.0,"last_snapshot_bytes":1937408,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963627,"remote_snapshot_timestamp":1726963627,"replay_state":"idle"}2024-09-22 00:10:05image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2820096.0,"last_snapshot_bytes":5476352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963632,"remote_snapshot_timestamp":1726963632,"replay_state":"idle"}2024-09-22 00:10:05image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26' 2024-09-22T00:10:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:10:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-22T00:10:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:10:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:10:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-22T00:10:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:10:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:26.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-22T00:10:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:10:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3309568.0,"last_snapshot_bytes":6455296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963625,"remote_snapshot_timestamp":1726963625,"replay_state":"idle"}2024-09-22 00:10:05image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1040384.0,"last_snapshot_bytes":1937408,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963627,"remote_snapshot_timestamp":1726963627,"replay_state":"idle"}2024-09-22 00:10:05image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2820096.0,"last_snapshot_bytes":5476352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963632,"remote_snapshot_timestamp":1726963632,"replay_state":"idle"}2024-09-22 00:10:05image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26' 2024-09-22T00:10:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:10:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3309568.0,"last_snapshot_bytes":6455296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963625,"remote_snapshot_timestamp":1726963625,"replay_state":"idle"}2024-09-22 00:10:05image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1040384.0,"last_snapshot_bytes":1937408,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963627,"remote_snapshot_timestamp":1726963627,"replay_state":"idle"}2024-09-22 00:10:05image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2820096.0,"last_snapshot_bytes":5476352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963632,"remote_snapshot_timestamp":1726963632,"replay_state":"idle"}2024-09-22 00:10:05image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26' 2024-09-22T00:10:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:10:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:10:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:10:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-22T00:10:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:10:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-22T00:10:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:10:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-22T00:10:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:10:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:10:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-22T00:10:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:10:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:10:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:10:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 859a2725-431b-4842-85c5-dceb1043514d 2024-09-22T00:10:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25 2024-09-22T00:10:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:10:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25' 2024-09-22T00:10:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:10:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: a1d90309-6eda-42f2-99de-fbe088cec238 2024-09-22T00:10:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25 2024-09-22T00:10:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:10:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25' 2024-09-22T00:10:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:10:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:10:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-22 00:10:25' 2024-09-22T00:10:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:10:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-22 00:10:25' 2024-09-22T00:10:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:10:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:10:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:25' 2024-09-22T00:10:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:25' 2024-09-22T00:10:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-22T00:10:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:10:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-22T00:10:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:10:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-22T00:10:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-22T00:10:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:10:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 52725bae-bfe7-4465-ad3b-965fe6fc62de 2024-09-22T00:10:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:10:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25 2024-09-22T00:10:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:10:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25' 2024-09-22T00:10:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:10:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-22 00:10:25' 2024-09-22T00:10:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:10:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:25' 2024-09-22T00:10:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:10:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: d5d01b87-f76d-4510-bda7-91bc59eb57b2 2024-09-22T00:10:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:10:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:26 2024-09-22T00:10:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:10:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25' 2024-09-22T00:10:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-22T00:10:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:10:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:27.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:27.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-22T00:10:27.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:10:27.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:27.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:27.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:27.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:27.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:26' 2024-09-22T00:10:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-22 00:10:26' 2024-09-22T00:10:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:10:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:26' 2024-09-22T00:10:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-22T00:10:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:10:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-22T00:10:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-22T00:10:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:10:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-22T00:10:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-22T00:10:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:10:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-22T00:10:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-22T00:10:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:10:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-22T00:10:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-22T00:10:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:10:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-22T00:10:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26' 2024-09-22T00:10:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:10:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:10:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:10:25cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:10:26' 2024-09-22T00:10:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:10:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:10:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-22T00:10:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:10:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-22T00:10:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:10:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:10:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:10:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:10:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 183dbf3b-282a-4c6d-9ed1-e9865ba1f6db 2024-09-22T00:10:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:26 2024-09-22T00:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25' 2024-09-22T00:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:10:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:26' 2024-09-22T00:10:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-22 00:10:26' 2024-09-22T00:10:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:10:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:26' 2024-09-22T00:10:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:10:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 7d09dfcc-3937-45be-b0b7-d2da1ba52854 2024-09-22T00:10:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:10:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:26 2024-09-22T00:10:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:10:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25' 2024-09-22T00:10:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:10:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: ffe30d9d-5e74-4ba9-81a4-1f3439aea154 2024-09-22T00:10:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:10:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:26 2024-09-22T00:10:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:10:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25' 2024-09-22T00:10:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:10:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:10:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 7d3beb22-c450-46bd-9c24-aa8184e646ef 2024-09-22T00:10:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:10:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:26 2024-09-22T00:10:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:10:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25' 2024-09-22T00:10:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:10:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: a21c9573-e1b1-4890-8240-0bb25f6afb4e 2024-09-22T00:10:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:10:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:26 2024-09-22T00:10:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:10:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25' 2024-09-22T00:10:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:27.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:10:27.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:27.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:27.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:27.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:27.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:27.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:27.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:27.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:27.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:27.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:27.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:27.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:10:27.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:27.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:27.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:10:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8fb6c676-c05c-4963-b930-bfc650676205 2024-09-22T00:10:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:10:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:26 2024-09-22T00:10:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:10:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25' 2024-09-22T00:10:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:27.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:27.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:27.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:26' 2024-09-22T00:10:27.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-22 00:10:26' 2024-09-22T00:10:27.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:27.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:10:27.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:26' 2024-09-22T00:10:27.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:10:27.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:27.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:27.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:27.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:26' 2024-09-22T00:10:27.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-22 00:10:26' 2024-09-22T00:10:27.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:10:27.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:26' 2024-09-22T00:10:27.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:27.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:27.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:27.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:27.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:10:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:27.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:26' 2024-09-22T00:10:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-22 00:10:26' 2024-09-22T00:10:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:10:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:26' 2024-09-22T00:10:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:26' 2024-09-22T00:10:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-22 00:10:26' 2024-09-22T00:10:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:10:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:26' 2024-09-22T00:10:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:26' 2024-09-22T00:10:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-22 00:10:26' 2024-09-22T00:10:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:10:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:26' 2024-09-22T00:10:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25' 2024-09-22T00:10:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:10:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:26' 2024-09-22T00:10:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:10:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-22T00:10:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:10:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:10:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-22T00:10:27.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25' 2024-09-22T00:10:27.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:10:27.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:26' 2024-09-22T00:10:27.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:10:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25' 2024-09-22T00:10:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:10:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:26' 2024-09-22T00:10:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:10:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-22T00:10:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:10:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:10:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-22T00:10:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-22T00:10:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:10:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:10:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-22T00:10:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25' 2024-09-22T00:10:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:10:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:26' 2024-09-22T00:10:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:10:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-22T00:10:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:10:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:10:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-22T00:10:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25' 2024-09-22T00:10:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:10:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25' 2024-09-22T00:10:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:10:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:26' 2024-09-22T00:10:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:10:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:26' 2024-09-22T00:10:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:10:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25' 2024-09-22T00:10:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-22T00:10:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:10:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:10:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-22T00:10:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-22T00:10:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:10:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:10:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-22T00:10:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:10:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:26' 2024-09-22T00:10:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:10:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25' 2024-09-22T00:10:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:10:28.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:10:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-22T00:10:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:10:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:10:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-22T00:10:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-22T00:10:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:10:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:10:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-22T00:10:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-22T00:10:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:10:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-22T00:10:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-22T00:10:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:10:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:10:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:10:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-22T00:10:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:10:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 859a2725-431b-4842-85c5-dceb1043514d 2024-09-22T00:10:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25 2024-09-22T00:10:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:10:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25' 2024-09-22T00:10:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:10:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: a1d90309-6eda-42f2-99de-fbe088cec238 2024-09-22T00:10:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:10:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25 2024-09-22T00:10:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:10:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:10:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:10:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:10:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:10:25' 2024-09-22T00:10:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:10:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:10:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:10:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:10:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:10:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:10:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:10:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:10:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:10:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:10:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:10:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-22 00:10:25' 2024-09-22T00:10:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:10:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:25' 2024-09-22T00:10:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-22 00:10:25' 2024-09-22T00:10:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:10:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:10:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:10:25' 2024-09-22T00:10:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:10:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:10:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:10:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25' 2024-09-22T00:10:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary4859a2725-431b-4842-85c5-dceb1043514dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary68fb6c676-c05c-4963-b930-bfc650676205up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:10:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:10:26cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:10:25' 2024-09-22T00:10:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:10:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:10:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:10:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:10:25' 2024-09-22T00:10:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:10:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:10:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:10:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:10:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:10:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:10:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-22T00:10:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:10:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:10:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:10:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-22T00:10:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-22T00:10:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:10:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:10:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:10:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:10:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-22T00:10:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:10:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:10:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:10:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:10:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:10:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:10:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:10:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:10:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:10:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:10:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-22T00:10:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:10:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:10:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-22T00:10:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:10:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:10:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-22T00:10:28.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:10:28.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:10:28.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-22T00:10:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:10:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:10:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-22T00:10:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:10:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:10:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:10:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:10:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-22T00:10:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-22T00:10:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:10:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:10:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-22T00:10:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd0 2024-09-22T00:10:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-22T00:10:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:10:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd3 2024-09-22T00:10:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd4 2024-09-22T00:10:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd1 2024-09-22T00:10:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-22T00:10:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-22T00:10:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:10:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:10:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:10:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-22T00:10:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd2 2024-09-22T00:10:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-22T00:10:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:10:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd5 2024-09-22T00:10:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-22T00:10:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:10:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd6 2024-09-22T00:10:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-22T00:10:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:10:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd7 2024-09-22T00:10:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-22T00:10:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:10:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:10:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:10:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:10:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:10:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-22T00:10:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:10:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:10:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-22T00:10:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd8 2024-09-22T00:10:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-22T00:10:30.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:10:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd9 2024-09-22T00:10:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-22T00:10:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:11:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=93808609080059515c367279f8edacac 2024-09-22T00:11:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-22T00:11:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 93808609080059515c367279f8edacac != \9\3\8\0\8\6\0\9\0\8\0\0\5\9\5\1\5\c\3\6\7\2\7\9\f\8\e\d\a\c\a\c ]] 2024-09-22T00:11:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:11:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 60875 2024-09-22T00:11:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=eca056fbd0ece1703df150eedccfbb6f 2024-09-22T00:11:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-22T00:11:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=fc1e139239a8dc8e2cca81fb1b132b9f 2024-09-22T00:11:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-22T00:11:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ fc1e139239a8dc8e2cca81fb1b132b9f != \f\c\1\e\1\3\9\2\3\9\a\8\d\c\8\e\2\c\c\a\8\1\f\b\1\b\1\3\2\b\9\f ]] 2024-09-22T00:11:02.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ eca056fbd0ece1703df150eedccfbb6f != \e\c\a\0\5\6\f\b\d\0\e\c\e\1\7\0\3\d\f\1\5\0\e\e\d\c\c\f\b\b\6\f ]] 2024-09-22T00:11:02.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=3607fb0bf757e3b27bab00c3f777e0ee 2024-09-22T00:11:02.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-22T00:11:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=7a8dbb279ae969377e30805ab10c4116 2024-09-22T00:11:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-22T00:11:02.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 3607fb0bf757e3b27bab00c3f777e0ee != \3\6\0\7\f\b\0\b\f\7\5\7\e\3\b\2\7\b\a\b\0\0\c\3\f\7\7\7\e\0\e\e ]] 2024-09-22T00:11:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=bb8a4d4ab16fa7465fd09204aeb12064 2024-09-22T00:11:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-22T00:11:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 7a8dbb279ae969377e30805ab10c4116 != \7\a\8\d\b\b\2\7\9\a\e\9\6\9\3\7\7\e\3\0\8\0\5\a\b\1\0\c\4\1\1\6 ]] 2024-09-22T00:11:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:11:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 60877 2024-09-22T00:11:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:11:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 60879 2024-09-22T00:11:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=e35e5ac5d82da35687f456b8c44c3c2c 2024-09-22T00:11:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-22T00:11:03.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ bb8a4d4ab16fa7465fd09204aeb12064 != \b\b\8\a\4\d\4\a\b\1\6\f\a\7\4\6\5\f\d\0\9\2\0\4\a\e\b\1\2\0\6\4 ]] 2024-09-22T00:11:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ e35e5ac5d82da35687f456b8c44c3c2c != \e\3\5\e\5\a\c\5\d\8\2\d\a\3\5\6\8\7\f\4\5\6\b\8\c\4\4\c\3\c\2\c ]] 2024-09-22T00:11:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=c5e431ea57f6b398b4d07fbf3231b363 2024-09-22T00:11:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-22T00:11:03.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ c5e431ea57f6b398b4d07fbf3231b363 != \c\5\e\4\3\1\e\a\5\7\f\6\b\3\9\8\b\4\d\0\7\f\b\f\3\2\3\1\b\3\6\3 ]] 2024-09-22T00:11:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=28f395986814b4292f0971e1b301a109 2024-09-22T00:11:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-22T00:11:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 28f395986814b4292f0971e1b301a109 != \2\8\f\3\9\5\9\8\6\8\1\4\b\4\2\9\2\f\0\9\7\1\e\1\b\3\0\1\a\1\0\9 ]] 2024-09-22T00:11:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:11:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 60881 2024-09-22T00:11:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:11:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 60883 2024-09-22T00:11:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:11:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 60885 2024-09-22T00:11:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=8b73c40b9791bdf7ee60d99993325d95 2024-09-22T00:11:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-22T00:11:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 8b73c40b9791bdf7ee60d99993325d95 != \8\b\7\3\c\4\0\b\9\7\9\1\b\d\f\7\e\e\6\0\d\9\9\9\9\3\3\2\5\d\9\5 ]] 2024-09-22T00:11:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:11:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 60886 2024-09-22T00:11:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:11:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 60888 2024-09-22T00:11:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:11:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 60890 2024-09-22T00:11:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:11:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-22T00:11:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-22T00:11:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:11:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:11:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-22T00:11:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:11:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:11:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-22T00:11:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:11:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:11:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:11:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: ffe30d9d-5e74-4ba9-81a4-1f3439aea154 2024-09-22T00:11:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:11:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2078173.87,"bytes_per_snapshot":62345216.0,"last_snapshot_bytes":62345216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"} 2024-09-22T00:11:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:11:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:05 2024-09-22T00:11:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:11:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:11:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:11:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:11:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:00' 2024-09-22T00:11:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:11:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:11:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:11:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2078173.87,"bytes_per_snapshot":62345216.0,"last_snapshot_bytes":62345216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"} 2024-09-22T00:11:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:11:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:11:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:11:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:11:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:11:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:11:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-22 00:11:05' 2024-09-22T00:11:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:11:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:11:05' 2024-09-22T00:11:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:11:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1ffe30d9d-5e74-4ba9-81a4-1f3439aea154up+replayingreplaying, {"bytes_per_second":2078173.87,"bytes_per_snapshot":62345216.0,"last_snapshot_bytes":62345216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05image-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+replayingreplaying, {"bytes_per_second":2051140.27,"bytes_per_snapshot":61534208.0,"last_snapshot_bytes":61534208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+replayingreplaying, {"bytes_per_second":2126233.6,"bytes_per_snapshot":63787008.0,"last_snapshot_bytes":63787008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+replayingreplaying, {"bytes_per_second":2229998.93,"bytes_per_snapshot":66899968.0,"last_snapshot_bytes":66899968,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05image-primary4859a2725-431b-4842-85c5-dceb1043514dup+replayingreplaying, {"bytes_per_second":2243652.27,"bytes_per_snapshot":67309568.0,"last_snapshot_bytes":67309568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963829,"remote_snapshot_timestamp":1726963829,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+replayingreplaying, {"bytes_per_second":2091008.0,"bytes_per_snapshot":62730240.0,"last_snapshot_bytes":62730240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05image-primary68fb6c676-c05c-4963-b930-bfc650676205up+replayingreplaying, {"bytes_per_second":2046088.53,"bytes_per_snapshot":61382656.0,"last_snapshot_bytes":61382656,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+replayingreplaying, {"bytes_per_second":2062472.53,"bytes_per_snapshot":61874176.0,"last_snapshot_bytes":61874176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963829,"remote_snapshot_timestamp":1726963829,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+replayingreplaying, {"bytes_per_second":2042129.07,"bytes_per_snapshot":61263872.0,"last_snapshot_bytes":61263872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+replayingreplaying, {"bytes_per_second":2408311.47,"bytes_per_snapshot":72249344.0,"last_snapshot_bytes":72249344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05' 2024-09-22T00:11:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:11:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:11:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:11:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:11:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:11:05.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-22T00:11:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:11:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:11:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-22T00:11:05.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-22T00:11:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:11:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-22T00:11:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:11:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:11:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary1 2024-09-22T00:11:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary1 2024-09-22T00:11:07.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:07.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-22T00:11:07.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-22T00:11:07.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:11:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:11:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-22T00:11:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:11:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:11:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-22T00:11:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:11:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:11:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:11:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 183dbf3b-282a-4c6d-9ed1-e9865ba1f6db 2024-09-22T00:11:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:11:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2126233.6,"bytes_per_snapshot":63787008.0,"last_snapshot_bytes":63787008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"} 2024-09-22T00:11:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:11:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:05 2024-09-22T00:11:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:11:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:11:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:11:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:11:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:05' 2024-09-22T00:11:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:11:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:11:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:11:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2126233.6,"bytes_per_snapshot":63787008.0,"last_snapshot_bytes":63787008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"} 2024-09-22T00:11:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:11:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:11:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:11:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:11:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:11:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-22 00:11:05' 2024-09-22T00:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:11:05' 2024-09-22T00:11:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:11:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK942614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+replayingreplaying, {"bytes_per_second":2051140.27,"bytes_per_snapshot":61534208.0,"last_snapshot_bytes":61534208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05image-primary2183dbf3b-282a-4c6d-9ed1-e9865ba1f6dbup+replayingreplaying, {"bytes_per_second":2126233.6,"bytes_per_snapshot":63787008.0,"last_snapshot_bytes":63787008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+replayingreplaying, {"bytes_per_second":2229998.93,"bytes_per_snapshot":66899968.0,"last_snapshot_bytes":66899968,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05image-primary4859a2725-431b-4842-85c5-dceb1043514dup+replayingreplaying, {"bytes_per_second":2243652.27,"bytes_per_snapshot":67309568.0,"last_snapshot_bytes":67309568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963829,"remote_snapshot_timestamp":1726963829,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+replayingreplaying, {"bytes_per_second":2091008.0,"bytes_per_snapshot":62730240.0,"last_snapshot_bytes":62730240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05image-primary68fb6c676-c05c-4963-b930-bfc650676205up+replayingreplaying, {"bytes_per_second":2046088.53,"bytes_per_snapshot":61382656.0,"last_snapshot_bytes":61382656,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+replayingreplaying, {"bytes_per_second":2062472.53,"bytes_per_snapshot":61874176.0,"last_snapshot_bytes":61874176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963829,"remote_snapshot_timestamp":1726963829,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+replayingreplaying, {"bytes_per_second":2042129.07,"bytes_per_snapshot":61263872.0,"last_snapshot_bytes":61263872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+replayingreplaying, {"bytes_per_second":2408311.47,"bytes_per_snapshot":72249344.0,"last_snapshot_bytes":72249344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:05' 2024-09-22T00:11:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:11:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:11:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:11:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:11:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:11:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-22T00:11:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:11:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:11:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-22T00:11:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-22T00:11:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:11:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-22T00:11:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:11:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:11:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary2 2024-09-22T00:11:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary2 2024-09-22T00:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-22T00:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-22T00:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-22T00:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:11:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-22T00:11:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:11:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:11:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:11:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 7d3beb22-c450-46bd-9c24-aa8184e646ef 2024-09-22T00:11:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:11:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2229998.93,"bytes_per_snapshot":66899968.0,"last_snapshot_bytes":66899968,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"} 2024-09-22T00:11:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:11:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:05 2024-09-22T00:11:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:11:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:11:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:11:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:11:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:10' 2024-09-22T00:11:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:11:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:11:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:11:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2229998.93,"bytes_per_snapshot":66899968.0,"last_snapshot_bytes":66899968,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"} 2024-09-22T00:11:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:11:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:11:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:11:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:11:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:11:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:11:11.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:11.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-22 00:11:05' 2024-09-22T00:11:11.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:11:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:11:05' 2024-09-22T00:11:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:11:11.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK842614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+replayingreplaying, {"bytes_per_second":2051140.27,"bytes_per_snapshot":61534208.0,"last_snapshot_bytes":61534208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:10image-primary37d3beb22-c450-46bd-9c24-aa8184e646efup+replayingreplaying, {"bytes_per_second":2229998.93,"bytes_per_snapshot":66899968.0,"last_snapshot_bytes":66899968,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:10image-primary4859a2725-431b-4842-85c5-dceb1043514dup+replayingreplaying, {"bytes_per_second":2243652.27,"bytes_per_snapshot":67309568.0,"last_snapshot_bytes":67309568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963829,"remote_snapshot_timestamp":1726963829,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:10image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+replayingreplaying, {"bytes_per_second":2091008.0,"bytes_per_snapshot":62730240.0,"last_snapshot_bytes":62730240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:10image-primary68fb6c676-c05c-4963-b930-bfc650676205up+replayingreplaying, {"bytes_per_second":2046088.53,"bytes_per_snapshot":61382656.0,"last_snapshot_bytes":61382656,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:10image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+replayingreplaying, {"bytes_per_second":2062472.53,"bytes_per_snapshot":61874176.0,"last_snapshot_bytes":61874176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963829,"remote_snapshot_timestamp":1726963829,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:10image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+replayingreplaying, {"bytes_per_second":2042129.07,"bytes_per_snapshot":61263872.0,"last_snapshot_bytes":61263872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:10image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+replayingreplaying, {"bytes_per_second":2408311.47,"bytes_per_snapshot":72249344.0,"last_snapshot_bytes":72249344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:10' 2024-09-22T00:11:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:11:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:11:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:11:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:11:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:11:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-22T00:11:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:11:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:11:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-22T00:11:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-22T00:11:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:11:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-22T00:11:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:11:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:11:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary3 2024-09-22T00:11:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary3 2024-09-22T00:11:13.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:13.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-22T00:11:13.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-22T00:11:13.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:11:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:11:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-22T00:11:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:11:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:11:13.412 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:11:13.411+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T00:11:13.412 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:11:13.411+0000 7f54f2d95640 -1 reset not still connected to 0x5617fc92fe10 2024-09-22T00:11:13.413 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:11:13.411+0000 7f54f2d95640 -1 reset not still connected to 0x5617ffc9cd00 2024-09-22T00:11:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-22T00:11:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:11:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:11:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:11:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 859a2725-431b-4842-85c5-dceb1043514d 2024-09-22T00:11:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:11:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2243652.27,"bytes_per_snapshot":67309568.0,"last_snapshot_bytes":67309568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963829,"remote_snapshot_timestamp":1726963829,"replay_state":"idle"} 2024-09-22T00:11:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:11:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:05 2024-09-22T00:11:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:11:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:11:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:11:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:11:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:10' 2024-09-22T00:11:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:11:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:11:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:11:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2243652.27,"bytes_per_snapshot":67309568.0,"last_snapshot_bytes":67309568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963829,"remote_snapshot_timestamp":1726963829,"replay_state":"idle"} 2024-09-22T00:11:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:11:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:11:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:11:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:11:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:11:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:11:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-22 00:11:05' 2024-09-22T00:11:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:11:14.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:14.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:11:05' 2024-09-22T00:11:14.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:14.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:14.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:11:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK742614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+replayingreplaying, {"bytes_per_second":2051140.27,"bytes_per_snapshot":61534208.0,"last_snapshot_bytes":61534208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:10image-primary4859a2725-431b-4842-85c5-dceb1043514dup+replayingreplaying, {"bytes_per_second":2243652.27,"bytes_per_snapshot":67309568.0,"last_snapshot_bytes":67309568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963829,"remote_snapshot_timestamp":1726963829,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:10image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+replayingreplaying, {"bytes_per_second":2091008.0,"bytes_per_snapshot":62730240.0,"last_snapshot_bytes":62730240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:10image-primary68fb6c676-c05c-4963-b930-bfc650676205up+replayingreplaying, {"bytes_per_second":2046088.53,"bytes_per_snapshot":61382656.0,"last_snapshot_bytes":61382656,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:10image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+replayingreplaying, {"bytes_per_second":2062472.53,"bytes_per_snapshot":61874176.0,"last_snapshot_bytes":61874176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963829,"remote_snapshot_timestamp":1726963829,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:10image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+replayingreplaying, {"bytes_per_second":2042129.07,"bytes_per_snapshot":61263872.0,"last_snapshot_bytes":61263872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:10image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+replayingreplaying, {"bytes_per_second":2408311.47,"bytes_per_snapshot":72249344.0,"last_snapshot_bytes":72249344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:10' 2024-09-22T00:11:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:11:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:11:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:11:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:11:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:11:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-22T00:11:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:11:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:11:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-22T00:11:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-22T00:11:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:11:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-22T00:11:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:11:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:11:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary4 2024-09-22T00:11:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary4 2024-09-22T00:11:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-22T00:11:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-22T00:11:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:11:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:11:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-22T00:11:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:11:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:11:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-22T00:11:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:11:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 52725bae-bfe7-4465-ad3b-965fe6fc62de 2024-09-22T00:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2091008.0,"bytes_per_snapshot":62730240.0,"last_snapshot_bytes":62730240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"} 2024-09-22T00:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:05 2024-09-22T00:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:15' 2024-09-22T00:11:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:11:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:11:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:11:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2091008.0,"bytes_per_snapshot":62730240.0,"last_snapshot_bytes":62730240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"} 2024-09-22T00:11:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:11:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:11:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:11:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:11:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:11:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:11:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-22 00:11:05' 2024-09-22T00:11:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:11:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:11:05' 2024-09-22T00:11:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:11:17.417 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:11:17.415+0000 7f54f2d95640 -1 reset not still connected to 0x5617fb121860 2024-09-22T00:11:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK642614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+replayingreplaying, {"bytes_per_second":2051140.27,"bytes_per_snapshot":61534208.0,"last_snapshot_bytes":61534208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:15image-primary552725bae-bfe7-4465-ad3b-965fe6fc62deup+replayingreplaying, {"bytes_per_second":2091008.0,"bytes_per_snapshot":62730240.0,"last_snapshot_bytes":62730240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:15image-primary68fb6c676-c05c-4963-b930-bfc650676205up+replayingreplaying, {"bytes_per_second":2046088.53,"bytes_per_snapshot":61382656.0,"last_snapshot_bytes":61382656,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:15image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+replayingreplaying, {"bytes_per_second":2062472.53,"bytes_per_snapshot":61874176.0,"last_snapshot_bytes":61874176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963829,"remote_snapshot_timestamp":1726963829,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:15image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+replayingreplaying, {"bytes_per_second":2042129.07,"bytes_per_snapshot":61263872.0,"last_snapshot_bytes":61263872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:15image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+replayingreplaying, {"bytes_per_second":2408311.47,"bytes_per_snapshot":72249344.0,"last_snapshot_bytes":72249344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:15' 2024-09-22T00:11:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:11:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:11:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:11:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:11:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:11:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-22T00:11:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:11:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:11:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-22T00:11:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-22T00:11:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:11:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-22T00:11:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:11:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:11:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary5 2024-09-22T00:11:19.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:19.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary5 2024-09-22T00:11:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-22T00:11:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-22T00:11:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:11:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:11:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-22T00:11:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:11:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:11:20.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-22T00:11:20.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:20.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:11:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:11:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:11:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8fb6c676-c05c-4963-b930-bfc650676205 2024-09-22T00:11:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:11:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2046088.53,"bytes_per_snapshot":61382656.0,"last_snapshot_bytes":61382656,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"} 2024-09-22T00:11:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:11:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:05 2024-09-22T00:11:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:11:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:11:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:11:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:11:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:15' 2024-09-22T00:11:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:11:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:11:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:11:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2046088.53,"bytes_per_snapshot":61382656.0,"last_snapshot_bytes":61382656,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"} 2024-09-22T00:11:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:11:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:11:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:11:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:11:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:11:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:11:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-22 00:11:05' 2024-09-22T00:11:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:11:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:11:05' 2024-09-22T00:11:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:11:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK542614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+replayingreplaying, {"bytes_per_second":2051140.27,"bytes_per_snapshot":61534208.0,"last_snapshot_bytes":61534208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:15image-primary68fb6c676-c05c-4963-b930-bfc650676205up+replayingreplaying, {"bytes_per_second":2046088.53,"bytes_per_snapshot":61382656.0,"last_snapshot_bytes":61382656,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:15image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+replayingreplaying, {"bytes_per_second":2062472.53,"bytes_per_snapshot":61874176.0,"last_snapshot_bytes":61874176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963829,"remote_snapshot_timestamp":1726963829,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:15image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+replayingreplaying, {"bytes_per_second":2042129.07,"bytes_per_snapshot":61263872.0,"last_snapshot_bytes":61263872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:15image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+replayingreplaying, {"bytes_per_second":2408311.47,"bytes_per_snapshot":72249344.0,"last_snapshot_bytes":72249344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:15' 2024-09-22T00:11:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:11:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:11:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:11:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:11:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:11:20.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-22T00:11:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:11:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:11:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-22T00:11:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-22T00:11:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:11:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-22T00:11:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:11:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:11:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary6 2024-09-22T00:11:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary6 2024-09-22T00:11:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-22T00:11:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-22T00:11:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:11:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:11:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-22T00:11:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:11:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:11:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-22T00:11:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:11:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:23.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:23.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:11:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:11:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: a1d90309-6eda-42f2-99de-fbe088cec238 2024-09-22T00:11:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:11:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2062472.53,"bytes_per_snapshot":61874176.0,"last_snapshot_bytes":61874176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963829,"remote_snapshot_timestamp":1726963829,"replay_state":"idle"} 2024-09-22T00:11:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:11:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:05 2024-09-22T00:11:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:11:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:11:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:11:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:11:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:20' 2024-09-22T00:11:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:11:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:11:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:11:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2062472.53,"bytes_per_snapshot":61874176.0,"last_snapshot_bytes":61874176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963829,"remote_snapshot_timestamp":1726963829,"replay_state":"idle"} 2024-09-22T00:11:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:11:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:11:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:11:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:11:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:11:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:11:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-22 00:11:05' 2024-09-22T00:11:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:11:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:11:05' 2024-09-22T00:11:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:11:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK442614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+replayingreplaying, {"bytes_per_second":2051140.27,"bytes_per_snapshot":61534208.0,"last_snapshot_bytes":61534208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:20image-primary7a1d90309-6eda-42f2-99de-fbe088cec238up+replayingreplaying, {"bytes_per_second":2062472.53,"bytes_per_snapshot":61874176.0,"last_snapshot_bytes":61874176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963829,"remote_snapshot_timestamp":1726963829,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:20image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+replayingreplaying, {"bytes_per_second":2042129.07,"bytes_per_snapshot":61263872.0,"last_snapshot_bytes":61263872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:20image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+replayingreplaying, {"bytes_per_second":2408311.47,"bytes_per_snapshot":72249344.0,"last_snapshot_bytes":72249344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:20' 2024-09-22T00:11:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:11:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:11:23.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:23.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:11:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:11:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:11:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-22T00:11:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:11:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:11:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-22T00:11:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-22T00:11:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:11:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-22T00:11:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:11:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:11:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary7 2024-09-22T00:11:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary7 2024-09-22T00:11:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-22T00:11:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-22T00:11:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:11:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:11:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-22T00:11:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:11:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:11:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-22T00:11:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:11:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:11:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:11:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: d5d01b87-f76d-4510-bda7-91bc59eb57b2 2024-09-22T00:11:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:11:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2042129.07,"bytes_per_snapshot":61263872.0,"last_snapshot_bytes":61263872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"} 2024-09-22T00:11:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:11:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:05 2024-09-22T00:11:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:11:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:11:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:11:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:11:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:25' 2024-09-22T00:11:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:11:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:11:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:11:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2042129.07,"bytes_per_snapshot":61263872.0,"last_snapshot_bytes":61263872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"} 2024-09-22T00:11:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:11:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:11:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:11:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:11:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:11:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:11:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-22 00:11:05' 2024-09-22T00:11:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:11:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:11:05' 2024-09-22T00:11:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:11:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+replayingreplaying, {"bytes_per_second":2051140.27,"bytes_per_snapshot":61534208.0,"last_snapshot_bytes":61534208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:25image-primary8d5d01b87-f76d-4510-bda7-91bc59eb57b2up+replayingreplaying, {"bytes_per_second":2042129.07,"bytes_per_snapshot":61263872.0,"last_snapshot_bytes":61263872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:25image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+replayingreplaying, {"bytes_per_second":2408311.47,"bytes_per_snapshot":72249344.0,"last_snapshot_bytes":72249344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:25' 2024-09-22T00:11:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:11:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:11:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:11:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:11:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:11:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-22T00:11:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:11:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:11:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-22T00:11:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-22T00:11:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:11:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-22T00:11:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:11:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:11:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary8 2024-09-22T00:11:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary8 2024-09-22T00:11:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-22T00:11:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-22T00:11:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:11:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:11:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-22T00:11:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:11:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:11:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-22T00:11:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:11:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:11:29.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:11:29.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: a21c9573-e1b1-4890-8240-0bb25f6afb4e 2024-09-22T00:11:29.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:11:29.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2408311.47,"bytes_per_snapshot":72249344.0,"last_snapshot_bytes":72249344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"} 2024-09-22T00:11:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:11:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:05 2024-09-22T00:11:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:11:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:11:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:11:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:11:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:25' 2024-09-22T00:11:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:11:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:11:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:11:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2408311.47,"bytes_per_snapshot":72249344.0,"last_snapshot_bytes":72249344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"} 2024-09-22T00:11:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:11:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:11:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:11:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:11:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:11:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:11:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-22 00:11:05' 2024-09-22T00:11:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:11:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:11:05' 2024-09-22T00:11:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:11:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK242614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+replayingreplaying, {"bytes_per_second":2051140.27,"bytes_per_snapshot":61534208.0,"last_snapshot_bytes":61534208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:25image-primary9a21c9573-e1b1-4890-8240-0bb25f6afb4eup+replayingreplaying, {"bytes_per_second":2408311.47,"bytes_per_snapshot":72249344.0,"last_snapshot_bytes":72249344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:25' 2024-09-22T00:11:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:11:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:11:29.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:29.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:11:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:11:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:11:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-22T00:11:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:11:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:11:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-22T00:11:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-22T00:11:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:11:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-22T00:11:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:11:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:11:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary9 2024-09-22T00:11:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary9 2024-09-22T00:11:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-22T00:11:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-22T00:11:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:11:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:11:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-22T00:11:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:31.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:31.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:11:31.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:31.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:11:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-22T00:11:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:11:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:11:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:11:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:11:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:11:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 7d09dfcc-3937-45be-b0b7-d2da1ba52854 2024-09-22T00:11:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:11:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2051140.27,"bytes_per_snapshot":61534208.0,"last_snapshot_bytes":61534208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"} 2024-09-22T00:11:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:11:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:05 2024-09-22T00:11:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:11:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:11:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:11:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:11:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:11:30' 2024-09-22T00:11:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:11:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:11:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:11:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2051140.27,"bytes_per_snapshot":61534208.0,"last_snapshot_bytes":61534208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"} 2024-09-22T00:11:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:11:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:11:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:11:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:11:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:11:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:11:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-22 00:11:05' 2024-09-22T00:11:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:11:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:11:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:11:05' 2024-09-22T00:11:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:11:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:11:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:11:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK142614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary107d09dfcc-3937-45be-b0b7-d2da1ba52854up+replayingreplaying, {"bytes_per_second":2051140.27,"bytes_per_snapshot":61534208.0,"last_snapshot_bytes":61534208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726963828,"remote_snapshot_timestamp":1726963828,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:11:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:11:30' 2024-09-22T00:11:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:11:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:11:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:11:05' 2024-09-22T00:11:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:11:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:11:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:11:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:11:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:11:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-22T00:11:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:11:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:11:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-22T00:11:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-22T00:11:33.339 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:11:33.339+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7c58148f0 2024-09-22T00:11:33.339 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:11:33.339+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7c5814d00 2024-09-22T00:11:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:11:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-22T00:11:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:11:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:11:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:11:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary10 2024-09-22T00:11:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary10 2024-09-22T00:11:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..10} 2024-09-22T00:11:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS=() 2024-09-22T00:11:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS=() 2024-09-22T00:11:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS=() 2024-09-22T00:11:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS=() 2024-09-22T00:11:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ RET=0 2024-09-22T00:11:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-22T00:11:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary1 2024-09-22T00:11:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-22T00:11:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:11:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:11:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-22T00:11:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:11:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:11:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:11:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:11:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-22T00:11:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:11:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:11:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-22T00:11:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:11:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-22T00:11:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:11:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-22T00:11:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary1': 2024-09-22T00:11:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:11:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:11:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:11:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1e2f6580b96 2024-09-22T00:11:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1e2f6580b96 2024-09-22T00:11:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:11:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:11:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:11:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:11:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:11:34 2024 2024-09-22T00:11:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:11:34 2024 2024-09-22T00:11:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:11:34 2024 2024-09-22T00:11:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:11:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:11:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 1efac4d0-57a5-4f9c-ad45-0595686a4e3b 2024-09-22T00:11:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:11:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:11:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:11:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-22T00:11:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd0 2024-09-22T00:11:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:11:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-22T00:11:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:11:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:11:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:11:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: c9478bfa-bb35-4e09-b1cb-c52b18dffcdf 2024-09-22T00:11:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:11:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:11:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:11:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:11:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:11:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:11:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary1 2024-09-22T00:11:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-22T00:11:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:11:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary1 2024-09-22T00:11:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:11:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-22T00:11:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:11:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-22T00:11:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary2 2024-09-22T00:11:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-22T00:11:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:11:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:11:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:11:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-22T00:11:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:11:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:11:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:11:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary1 2024-09-22T00:11:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:11:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:11:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-22T00:11:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:11:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:11:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary1 2024-09-22T00:11:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-22T00:11:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:11:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:11:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-22T00:11:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:11:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-22T00:11:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 995 2024-09-22T00:11:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:11:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-22T00:11:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary2': 2024-09-22T00:11:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:11:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:11:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:11:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1e3f30e27275 2024-09-22T00:11:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1e3f30e27275 2024-09-22T00:11:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:11:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:11:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:11:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:11:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:11:35 2024 2024-09-22T00:11:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:11:35 2024 2024-09-22T00:11:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:11:35 2024 2024-09-22T00:11:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:11:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:11:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 46a0a802-96d3-4bb0-bdb7-f572fb64611f 2024-09-22T00:11:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:11:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:11:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:11:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-22T00:11:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd1 2024-09-22T00:11:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:11:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-22T00:11:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:11:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:11:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:11:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 05f01ea4-7aec-4332-8c5d-496e2b17ba45 2024-09-22T00:11:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:11:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:11:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:11:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:11:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:11:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:11:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary2 2024-09-22T00:11:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-22T00:11:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:11:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-22T00:11:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:11:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary2 2024-09-22T00:11:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:11:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-22T00:11:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary2 2024-09-22T00:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-22T00:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary3 2024-09-22T00:11:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-22T00:11:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary2 2024-09-22T00:11:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-22T00:11:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:11:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:11:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-22T00:11:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:11:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:11:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:11:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:11:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-22T00:11:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:11:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:11:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-22T00:11:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:11:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-22T00:11:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:11:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-22T00:11:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:11:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 997 2024-09-22T00:11:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-22T00:11:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary3': 2024-09-22T00:11:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:11:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:11:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:11:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1e4e4f78354f 2024-09-22T00:11:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1e4e4f78354f 2024-09-22T00:11:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:11:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:11:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:11:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:11:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:11:36 2024 2024-09-22T00:11:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:11:36 2024 2024-09-22T00:11:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:11:36 2024 2024-09-22T00:11:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:11:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:11:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: f74c1283-be68-450d-aa81-c2905ec3b6bd 2024-09-22T00:11:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:11:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:11:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:11:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-22T00:11:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd2 2024-09-22T00:11:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:11:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-22T00:11:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:11:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:11:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:11:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: a02531da-35a0-4694-b0ac-6277428b21ed 2024-09-22T00:11:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:11:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:11:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:11:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:11:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:11:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:11:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary3 2024-09-22T00:11:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-22T00:11:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:11:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary3 2024-09-22T00:11:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:11:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-22T00:11:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:11:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:11:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-22T00:11:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary4 2024-09-22T00:11:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary3 2024-09-22T00:11:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-22T00:11:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:11:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:11:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:11:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:11:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:11:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-22T00:11:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary3 2024-09-22T00:11:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:11:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:11:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:11:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-22T00:11:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:11:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-22T00:11:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:11:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:11:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-22T00:11:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:11:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-22T00:11:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:11:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-22T00:11:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 999 2024-09-22T00:11:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:11:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:38.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-22T00:11:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary4': 2024-09-22T00:11:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:11:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:11:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:11:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1e5d9e96e07d 2024-09-22T00:11:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1e5d9e96e07d 2024-09-22T00:11:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:11:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:11:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:11:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:11:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:11:37 2024 2024-09-22T00:11:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:11:37 2024 2024-09-22T00:11:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:11:37 2024 2024-09-22T00:11:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:11:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:11:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 57679ffa-22a0-4454-9dd1-24f459258a36 2024-09-22T00:11:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:11:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:11:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:11:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-22T00:11:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd3 2024-09-22T00:11:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:11:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-22T00:11:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:11:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:11:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:11:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: f9787de5-6e9e-462c-9e1b-842b0cd38707 2024-09-22T00:11:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:11:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:11:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:11:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:11:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:11:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:11:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary4 2024-09-22T00:11:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-22T00:11:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:11:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary4 2024-09-22T00:11:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:11:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-22T00:11:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:11:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:11:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:11:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:11:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:11:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-22T00:11:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary5 2024-09-22T00:11:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-22T00:11:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary4 2024-09-22T00:11:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:11:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:11:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-22T00:11:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:11:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:11:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:11:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:11:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-22T00:11:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary4 2024-09-22T00:11:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-22T00:11:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:11:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:11:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-22T00:11:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:11:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-22T00:11:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:11:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-22T00:11:38.902 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:11:38.900+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a3630960d0 2024-09-22T00:11:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1001 2024-09-22T00:11:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:11:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:11:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:11:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:11:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-22T00:11:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary5': 2024-09-22T00:11:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:11:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:11:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:11:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1e6ca9136fc3 2024-09-22T00:11:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1e6ca9136fc3 2024-09-22T00:11:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:11:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:11:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:11:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:11:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:11:38 2024 2024-09-22T00:11:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:11:38 2024 2024-09-22T00:11:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:11:38 2024 2024-09-22T00:11:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:11:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:11:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9cf9377c-212a-418d-80e7-6b582738ce1e 2024-09-22T00:11:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:11:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:11:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:11:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-22T00:11:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd4 2024-09-22T00:11:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:11:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-22T00:11:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:11:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:11:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:11:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: d0eab354-ebda-45d2-9591-1471b52bdf02 2024-09-22T00:11:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:11:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:11:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:11:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:11:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:11:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:11:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary5 2024-09-22T00:11:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-22T00:11:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:11:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary5 2024-09-22T00:11:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:11:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:11:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:11:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:11:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-22T00:11:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:11:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:11:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:11:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:11:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:11:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary5 2024-09-22T00:11:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-22T00:11:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary6 2024-09-22T00:11:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-22T00:11:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary5 2024-09-22T00:11:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-22T00:11:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:11:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:11:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-22T00:11:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:11:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:11:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:11:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:11:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-22T00:11:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:11:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:11:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-22T00:11:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:11:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:40.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-22T00:11:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:11:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-22T00:11:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1003 2024-09-22T00:11:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:11:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1005 2024-09-22T00:11:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-22T00:11:41.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary6': 2024-09-22T00:11:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:11:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:11:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:11:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1e81cae9e21d 2024-09-22T00:11:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1e81cae9e21d 2024-09-22T00:11:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:11:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:11:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:11:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:11:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:11:40 2024 2024-09-22T00:11:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:11:40 2024 2024-09-22T00:11:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:11:40 2024 2024-09-22T00:11:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:11:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:11:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: d3b5b5e9-88cd-4138-ba00-75ea869fb1dd 2024-09-22T00:11:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:11:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:11:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:11:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-22T00:11:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd5 2024-09-22T00:11:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:11:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-22T00:11:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:11:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:11:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:11:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:11:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:11:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:11:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: f61e41e8-c71c-431e-9f02-712cf60f103e 2024-09-22T00:11:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:11:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:11:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:11:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:11:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:11:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:11:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary6 2024-09-22T00:11:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-22T00:11:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:11:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary6 2024-09-22T00:11:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:11:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-22T00:11:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:11:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary6 2024-09-22T00:11:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-22T00:11:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary7 2024-09-22T00:11:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-22T00:11:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary6 2024-09-22T00:11:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-22T00:11:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:11:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:11:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:11:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-22T00:11:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:11:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:11:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:11:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:11:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:11:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:11:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:11:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-22T00:11:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:11:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:11:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-22T00:11:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:11:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-22T00:11:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:11:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-22T00:11:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1004 2024-09-22T00:11:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:11:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:11:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:11:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:11:42.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1008 2024-09-22T00:11:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-22T00:11:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary7': 2024-09-22T00:11:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:11:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:11:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:11:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1e929f6ad774 2024-09-22T00:11:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1e929f6ad774 2024-09-22T00:11:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:11:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:11:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:11:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:11:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:11:41 2024 2024-09-22T00:11:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:11:41 2024 2024-09-22T00:11:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:11:41 2024 2024-09-22T00:11:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:11:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:11:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: f038f20d-309b-496d-b976-53bbf0460858 2024-09-22T00:11:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:11:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:11:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:11:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-22T00:11:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd6 2024-09-22T00:11:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:11:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-22T00:11:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:11:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:11:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:11:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 689cfb15-e929-4297-9636-09411d25435a 2024-09-22T00:11:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:11:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:11:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:11:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:11:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:11:42.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:11:42.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary7 2024-09-22T00:11:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-22T00:11:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:11:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary7 2024-09-22T00:11:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:11:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-22T00:11:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:11:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:11:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:11:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:11:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:11:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-22T00:11:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary8 2024-09-22T00:11:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-22T00:11:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary7 2024-09-22T00:11:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:11:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:11:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary7 2024-09-22T00:11:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-22T00:11:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-22T00:11:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:11:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:11:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:11:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:11:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-22T00:11:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:11:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:11:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-22T00:11:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:11:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-22T00:11:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:11:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-22T00:11:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1007 2024-09-22T00:11:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:11:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-22T00:11:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1011 2024-09-22T00:11:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary8': 2024-09-22T00:11:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:11:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:11:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:11:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1ea36c102514 2024-09-22T00:11:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1ea36c102514 2024-09-22T00:11:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:11:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:11:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:11:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:11:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:11:42 2024 2024-09-22T00:11:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:11:42 2024 2024-09-22T00:11:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:11:42 2024 2024-09-22T00:11:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:11:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:11:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 85b93c29-7d43-4d0a-8feb-205586f1412b 2024-09-22T00:11:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:11:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:11:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:11:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-22T00:11:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd7 2024-09-22T00:11:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:11:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-22T00:11:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:11:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:11:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:11:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 9b1ff908-b364-4239-8404-bed2aa9aad9b 2024-09-22T00:11:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:11:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:11:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:11:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:11:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:11:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:11:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary8 2024-09-22T00:11:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-22T00:11:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:11:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary8 2024-09-22T00:11:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:11:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-22T00:11:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:11:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:11:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:11:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:11:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:11:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-22T00:11:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary9 2024-09-22T00:11:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary8 2024-09-22T00:11:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-22T00:11:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:11:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:11:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-22T00:11:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary8 2024-09-22T00:11:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-22T00:11:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:11:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:11:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:11:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:11:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-22T00:11:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:11:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:11:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-22T00:11:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:11:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-22T00:11:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:11:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-22T00:11:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1010 2024-09-22T00:11:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1013 2024-09-22T00:11:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:11:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:11:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:11:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:11:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-22T00:11:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:11:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:11:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:11:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary9': 2024-09-22T00:11:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:11:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:11:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:11:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1eb295c67bbf 2024-09-22T00:11:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1eb295c67bbf 2024-09-22T00:11:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:11:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:11:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:11:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:11:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:11:43 2024 2024-09-22T00:11:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:11:43 2024 2024-09-22T00:11:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:11:43 2024 2024-09-22T00:11:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:11:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:11:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: f3d555de-4bdb-4256-96ae-c858dd8364e9 2024-09-22T00:11:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:11:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:11:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:11:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-22T00:11:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd8 2024-09-22T00:11:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:11:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-22T00:11:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:11:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:11:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:11:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: c29dac06-44bb-438e-b4de-fcc93aa2ec90 2024-09-22T00:11:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:11:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:11:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:11:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:11:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:11:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:11:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary9 2024-09-22T00:11:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-22T00:11:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:11:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary9 2024-09-22T00:11:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:11:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-22T00:11:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:11:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:11:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:11:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary9 2024-09-22T00:11:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:11:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:11:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-22T00:11:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary10 2024-09-22T00:11:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:11:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-22T00:11:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary9 2024-09-22T00:11:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-22T00:11:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:11:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:11:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-22T00:11:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:11:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:11:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:11:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:11:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:11:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-22T00:11:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:11:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:11:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:11:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-22T00:11:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:11:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:11:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-22T00:11:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:11:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-22T00:11:44.617 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:11:44.615+0000 7f54f2d95640 -1 reset not still connected to 0x5617f5b151e0 2024-09-22T00:11:44.617 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:11:44.615+0000 7f54f2d95640 -1 reset not still connected to 0x5617f7e30c30 2024-09-22T00:11:44.618 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:11:44.619+0000 7f54f2d95640 -1 reset not still connected to 0x56180dfee750 2024-09-22T00:11:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:11:45.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:45.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:11:45.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:45.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:45.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:11:45.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:11:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1017 2024-09-22T00:11:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-22T00:11:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary10': 2024-09-22T00:11:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:11:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:11:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:11:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 1ec54cb57b05 2024-09-22T00:11:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.1ec54cb57b05 2024-09-22T00:11:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:11:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:11:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:11:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:11:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:11:44 2024 2024-09-22T00:11:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:11:44 2024 2024-09-22T00:11:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:11:44 2024 2024-09-22T00:11:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:11:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:11:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: b40c8172-0f71-438a-aaba-7aeb6f08c4a1 2024-09-22T00:11:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:11:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:11:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:11:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-22T00:11:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:11:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:11:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:11:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd9 2024-09-22T00:11:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:11:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-22T00:11:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:11:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:11:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:11:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 108ce94e-7adf-4803-b887-4a6586488212 2024-09-22T00:11:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:11:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:11:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:45.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:11:45.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:11:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:11:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:11:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:11:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary10 2024-09-22T00:11:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-22T00:11:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:11:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary10 2024-09-22T00:11:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:11:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-22T00:11:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:11:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:11:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:11:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:11:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:11:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary10 2024-09-22T00:11:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:11:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 64459 2024-09-22T00:11:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary10 2024-09-22T00:11:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-22T00:11:45.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:11:45.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-22T00:11:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1015 2024-09-22T00:11:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1016 2024-09-22T00:11:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:11:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:11:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:11:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1021 2024-09-22T00:11:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:11:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:11:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:11:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:11:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:11:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:11:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1020 2024-09-22T00:11:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:11:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:11:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:11:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1019 2024-09-22T00:11:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:47.307 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:11:47.304+0000 7fab1f496640 -1 reset not still connected to 0x55fa643060d0 2024-09-22T00:11:47.307 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:11:47.304+0000 7fab1f496640 -1 reset not still connected to 0x55fa690e7380 2024-09-22T00:11:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1023 2024-09-22T00:11:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1022 2024-09-22T00:11:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:11:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:11:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:11:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1024 2024-09-22T00:11:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1025 2024-09-22T00:11:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:11:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:11:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:11:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:11:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:11:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:11:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:11:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:11:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:11:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1026 2024-09-22T00:11:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:11:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:11:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:11:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:11:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:11:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:11:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1027 2024-09-22T00:11:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1029 2024-09-22T00:11:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1028 2024-09-22T00:11:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:11:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:11:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:11:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:11:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:11:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:11:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:11:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:11:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:11:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:11:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:11:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:11:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1031 2024-09-22T00:11:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1030 2024-09-22T00:11:52.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1032 2024-09-22T00:11:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1033 2024-09-22T00:11:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:11:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:11:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:11:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:53.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1035 2024-09-22T00:11:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1034 2024-09-22T00:11:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:11:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:11:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:11:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:11:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:11:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:11:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:11:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:11:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:11:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1036 2024-09-22T00:11:55.070 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:11:55.062+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7cbbffc70 2024-09-22T00:11:55.070 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:11:55.066+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7e540a410 2024-09-22T00:11:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:11:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:11:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:11:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:11:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:11:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:11:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1037 2024-09-22T00:11:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1038 2024-09-22T00:11:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1039 2024-09-22T00:11:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:11:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:11:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:11:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:11:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:11:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:11:56.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:11:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:11:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:11:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:11:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:11:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:11:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1041 2024-09-22T00:11:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1040 2024-09-22T00:11:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:57.154 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:11:57.154+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7cbbffc70 2024-09-22T00:11:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1043 2024-09-22T00:11:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:11:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:11:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:11:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1042 2024-09-22T00:11:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1045 2024-09-22T00:11:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1044 2024-09-22T00:11:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:11:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:11:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:11:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:11:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:59.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:11:59.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:59.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:59.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:11:59.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:11:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:11:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:11:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:11:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:11:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:11:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:12:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1046 2024-09-22T00:12:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:12:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:12:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:12:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:12:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:12:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:12:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:01.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1048 2024-09-22T00:12:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1049 2024-09-22T00:12:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1047 2024-09-22T00:12:01.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:01.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:12:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:12:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:12:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:12:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:12:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:12:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:01.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:12:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:12:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:12:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:12:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:12:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:12:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1051 2024-09-22T00:12:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1050 2024-09-22T00:12:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:02.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:03.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1053 2024-09-22T00:12:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1052 2024-09-22T00:12:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:12:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:12:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:12:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1055 2024-09-22T00:12:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1054 2024-09-22T00:12:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:12:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:12:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:12:04.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:12:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:12:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:12:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:12:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:12:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:12:05.156 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:12:05.154+0000 7f54f2d95640 -1 reset not still connected to 0x5617f6fc31e0 2024-09-22T00:12:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1056 2024-09-22T00:12:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:12:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:12:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:12:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:12:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:12:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:12:06.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1058 2024-09-22T00:12:06.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1057 2024-09-22T00:12:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1059 2024-09-22T00:12:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:12:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:12:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:12:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:12:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:12:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:12:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:12:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:12:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:12:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:12:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:12:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:12:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1061 2024-09-22T00:12:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1060 2024-09-22T00:12:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:07.251 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:12:07.250+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7d3d41ba0 2024-09-22T00:12:07.252 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:12:07.250+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7d4e7fad0 2024-09-22T00:12:08.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1062 2024-09-22T00:12:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1063 2024-09-22T00:12:08.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:12:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:12:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:12:08.219 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:12:08.218+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7e540a410 2024-09-22T00:12:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1064 2024-09-22T00:12:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1065 2024-09-22T00:12:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:12:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:12:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:12:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:12:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:12:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:12:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:12:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:12:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:12:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1066 2024-09-22T00:12:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:12:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:12:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:12:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:12:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:12:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:12:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1068 2024-09-22T00:12:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1069 2024-09-22T00:12:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1067 2024-09-22T00:12:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:12:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:12:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:12:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:12:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:12:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:12:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1070 2024-09-22T00:12:12.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1071 2024-09-22T00:12:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:12:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:12:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:12:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:12:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:12:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:12:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1073 2024-09-22T00:12:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1072 2024-09-22T00:12:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:12:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:12:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:12:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1075 2024-09-22T00:12:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1074 2024-09-22T00:12:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:12:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:12:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:12:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:12:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:12:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:12:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:12:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:12:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:12:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1076 2024-09-22T00:12:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:12:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:12:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:12:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:12:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:12:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:12:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1079 2024-09-22T00:12:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1078 2024-09-22T00:12:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1077 2024-09-22T00:12:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:12:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:12:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:12:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:12:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:12:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:12:16.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:16.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1080 2024-09-22T00:12:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:12:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:12:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:12:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1081 2024-09-22T00:12:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:12:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:12:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:12:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:17.163 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:12:17.156+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b32ea0 2024-09-22T00:12:17.163 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:12:17.156+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b332b0 2024-09-22T00:12:17.163 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:12:17.156+0000 7f84953c5640 -1 reset not still connected to 0x5636f8ec80d0 2024-09-22T00:12:17.163 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:12:17.156+0000 7f84953c5640 -1 reset not still connected to 0x5636f9c289c0 2024-09-22T00:12:17.163 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:12:17.156+0000 7f84953c5640 -1 reset not still connected to 0x5636fbc9dc70 2024-09-22T00:12:17.163 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:12:17.156+0000 7f84953c5640 -1 reset not still connected to 0x5636fcfb6c30 2024-09-22T00:12:17.163 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:12:17.156+0000 7f84953c5640 -1 reset not still connected to 0x56370159b1e0 2024-09-22T00:12:18.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1083 2024-09-22T00:12:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:12:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:18.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:12:18.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:12:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1082 2024-09-22T00:12:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:18.114 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:12:18.114+0000 7f54f2d95640 -1 reset not still connected to 0x5617f42c9450 2024-09-22T00:12:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1085 2024-09-22T00:12:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1084 2024-09-22T00:12:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:12:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:12:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:12:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:12:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:12:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:12:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:12:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:12:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:12:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1086 2024-09-22T00:12:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:12:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:12:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:12:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:12:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:12:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:12:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1088 2024-09-22T00:12:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1087 2024-09-22T00:12:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1089 2024-09-22T00:12:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:12:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:12:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:12:21.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:21.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:12:21.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:21.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:21.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:12:21.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:12:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:22.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:22.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:12:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:12:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:12:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:12:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:12:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:12:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1091 2024-09-22T00:12:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1090 2024-09-22T00:12:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:23.076 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:12:23.070+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7cbbffc70 2024-09-22T00:12:23.077 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:12:23.074+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7e540a410 2024-09-22T00:12:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:12:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:12:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:12:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1093 2024-09-22T00:12:23.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1092 2024-09-22T00:12:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1094 2024-09-22T00:12:24.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1095 2024-09-22T00:12:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:12:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:12:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:12:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:12:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:12:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:12:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:12:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:12:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:12:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:12:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:12:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:12:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:12:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:12:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:12:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1096 2024-09-22T00:12:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1097 2024-09-22T00:12:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1098 2024-09-22T00:12:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1099 2024-09-22T00:12:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:12:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:12:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:12:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:12:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:12:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:12:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:12:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:12:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:12:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:12:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:12:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:12:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1101 2024-09-22T00:12:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1100 2024-09-22T00:12:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1102 2024-09-22T00:12:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:12:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:12:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:12:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1103 2024-09-22T00:12:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:28.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1105 2024-09-22T00:12:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1104 2024-09-22T00:12:29.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:29.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:12:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:12:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:12:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:12:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:12:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:12:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:12:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:12:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:12:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1106 2024-09-22T00:12:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:12:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:12:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:12:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:12:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:12:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:12:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1107 2024-09-22T00:12:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1108 2024-09-22T00:12:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1109 2024-09-22T00:12:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:12:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:12:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:12:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:12:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:12:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:12:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:12:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:12:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:12:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1110 2024-09-22T00:12:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1111 2024-09-22T00:12:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1112 2024-09-22T00:12:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1113 2024-09-22T00:12:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:12:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:12:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:12:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1114 2024-09-22T00:12:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1115 2024-09-22T00:12:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:12:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:12:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:12:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:12:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:12:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:12:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:12:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:12:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:12:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1116 2024-09-22T00:12:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:12:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:12:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:12:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:12:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:12:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:12:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1119 2024-09-22T00:12:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1118 2024-09-22T00:12:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1117 2024-09-22T00:12:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:36.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:36.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:36.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:12:36.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:36.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:36.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:12:36.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:12:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:12:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:12:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:12:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1120 2024-09-22T00:12:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1121 2024-09-22T00:12:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:12:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:12:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:12:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:12:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:12:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:12:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1123 2024-09-22T00:12:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1122 2024-09-22T00:12:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:12:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:12:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:12:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1125 2024-09-22T00:12:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1124 2024-09-22T00:12:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:12:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:12:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:12:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:12:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:12:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:12:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:12:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:12:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1126 2024-09-22T00:12:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:12:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:12:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:12:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:12:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:12:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:12:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1129 2024-09-22T00:12:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1127 2024-09-22T00:12:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:12:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:12:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:12:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1128 2024-09-22T00:12:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:12:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:12:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:12:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1131 2024-09-22T00:12:42.038 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:12:42.034+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7c486b2b0 2024-09-22T00:12:42.038 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:12:42.034+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7c591e410 2024-09-22T00:12:42.038 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:12:42.034+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7c96831e0 2024-09-22T00:12:42.039 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:12:42.034+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7e540a410 2024-09-22T00:12:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:12:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:12:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:12:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1130 2024-09-22T00:12:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:12:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:12:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:12:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1133 2024-09-22T00:12:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1132 2024-09-22T00:12:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:12:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:12:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:12:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1134 2024-09-22T00:12:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1135 2024-09-22T00:12:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:12:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:12:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:12:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:12:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:12:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:12:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:12:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:12:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:12:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1136 2024-09-22T00:12:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:12:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:12:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:12:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:12:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:12:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:12:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1139 2024-09-22T00:12:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1138 2024-09-22T00:12:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1137 2024-09-22T00:12:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:12:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:12:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:12:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:12:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:12:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:12:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:12:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:12:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:12:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1140 2024-09-22T00:12:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:12:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:12:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:12:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1141 2024-09-22T00:12:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:47.192 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:12:47.190+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7d4e7fad0 2024-09-22T00:12:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1142 2024-09-22T00:12:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:12:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:12:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:12:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1143 2024-09-22T00:12:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1144 2024-09-22T00:12:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1145 2024-09-22T00:12:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:12:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:12:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:12:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:12:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:12:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:12:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:12:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:12:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:12:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1146 2024-09-22T00:12:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:12:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:12:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:12:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:12:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:12:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:12:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1148 2024-09-22T00:12:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1149 2024-09-22T00:12:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1147 2024-09-22T00:12:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:12:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:12:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:12:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:12:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:12:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:12:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:51.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:12:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:12:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:12:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:12:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:12:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:12:52.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1150 2024-09-22T00:12:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1151 2024-09-22T00:12:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1153 2024-09-22T00:12:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1152 2024-09-22T00:12:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:12:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:12:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:12:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1155 2024-09-22T00:12:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1154 2024-09-22T00:12:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:12:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:12:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:12:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:12:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:12:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:12:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1156 2024-09-22T00:12:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:12:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:12:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:12:55.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:55.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:12:55.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:55.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:55.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:12:55.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:12:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1158 2024-09-22T00:12:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1159 2024-09-22T00:12:56.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1157 2024-09-22T00:12:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:12:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:12:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:12:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:12:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:12:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:12:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:12:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:12:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:12:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:12:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:12:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:12:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1160 2024-09-22T00:12:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1161 2024-09-22T00:12:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1163 2024-09-22T00:12:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1162 2024-09-22T00:12:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:12:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:12:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:12:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1164 2024-09-22T00:12:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1165 2024-09-22T00:12:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:12:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:12:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:12:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:12:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:12:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:12:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:12:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:12:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:12:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:12:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:12:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:12:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:13:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1166 2024-09-22T00:13:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:13:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:13:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:13:00.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:13:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:13:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:13:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1169 2024-09-22T00:13:01.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1167 2024-09-22T00:13:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:13:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:13:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:13:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1168 2024-09-22T00:13:01.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:01.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:13:01.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:01.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:13:01.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:13:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:13:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:13:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:13:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:13:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:13:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:13:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1170 2024-09-22T00:13:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1171 2024-09-22T00:13:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1172 2024-09-22T00:13:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1173 2024-09-22T00:13:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:13:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:13:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:13:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1175 2024-09-22T00:13:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1174 2024-09-22T00:13:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:13:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:13:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:13:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:13:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:13:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:13:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:13:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:13:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:13:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1176 2024-09-22T00:13:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:13:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:13:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:13:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:13:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:13:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:13:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1178 2024-09-22T00:13:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1177 2024-09-22T00:13:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1179 2024-09-22T00:13:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:13:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:13:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:13:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:13:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:13:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:13:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1180 2024-09-22T00:13:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1181 2024-09-22T00:13:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:13:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:13:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:13:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:13:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:13:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:13:07.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1182 2024-09-22T00:13:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1183 2024-09-22T00:13:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:13:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:13:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:13:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1184 2024-09-22T00:13:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1185 2024-09-22T00:13:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:13:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:13:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:13:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:13:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:13:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:13:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:13:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:13:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:13:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1186 2024-09-22T00:13:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:13:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:13:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:13:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:13:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:13:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:13:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1187 2024-09-22T00:13:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1188 2024-09-22T00:13:11.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1189 2024-09-22T00:13:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:13:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:13:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:13:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:13:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:13:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:13:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:11.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1191 2024-09-22T00:13:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1190 2024-09-22T00:13:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:13:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:13:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:13:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:13:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:13:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:13:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1192 2024-09-22T00:13:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1193 2024-09-22T00:13:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:13:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:13:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:13:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1194 2024-09-22T00:13:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1195 2024-09-22T00:13:14.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:13:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:13:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:13:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:13:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:13:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:13:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:13:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:13:14.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:13:15.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1196 2024-09-22T00:13:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:13:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:13:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:13:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:13:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:13:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:13:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:15.467 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:13:15.465+0000 7f54f2d95640 -1 reset not still connected to 0x5617f5b151e0 2024-09-22T00:13:15.468 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:13:15.465+0000 7f54f2d95640 -1 reset not still connected to 0x5617f7e30c30 2024-09-22T00:13:15.468 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:13:15.465+0000 7f54f2d95640 -1 reset not still connected to 0x56180dfee750 2024-09-22T00:13:16.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1197 2024-09-22T00:13:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1198 2024-09-22T00:13:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1199 2024-09-22T00:13:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:13:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:13:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:13:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:13:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:13:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:13:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:16.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:13:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:13:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:13:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:13:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:13:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:13:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1201 2024-09-22T00:13:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1200 2024-09-22T00:13:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1202 2024-09-22T00:13:18.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:18.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:13:18.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:18.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:18.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:13:18.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:13:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1203 2024-09-22T00:13:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1204 2024-09-22T00:13:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1205 2024-09-22T00:13:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:13:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:13:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:13:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:13:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:13:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:13:19.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:19.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:13:19.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:13:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:13:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1206 2024-09-22T00:13:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:13:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:13:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:13:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:13:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:13:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:13:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1207 2024-09-22T00:13:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1208 2024-09-22T00:13:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1209 2024-09-22T00:13:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:13:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:13:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:13:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:13:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:13:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:13:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:13:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:13:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:13:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:13:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:13:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:13:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1210 2024-09-22T00:13:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1211 2024-09-22T00:13:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1212 2024-09-22T00:13:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1213 2024-09-22T00:13:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:13:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:13:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:13:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1214 2024-09-22T00:13:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1215 2024-09-22T00:13:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:13:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:13:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:13:24.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:24.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:13:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:13:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:13:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:13:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:13:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:13:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1216 2024-09-22T00:13:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:13:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:13:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:13:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:13:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:13:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:13:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1219 2024-09-22T00:13:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1218 2024-09-22T00:13:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1217 2024-09-22T00:13:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:13:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:13:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:13:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:13:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:13:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:13:26.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:27.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:13:27.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:27.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:27.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:13:27.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:13:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:13:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:13:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:13:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1221 2024-09-22T00:13:27.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1220 2024-09-22T00:13:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:13:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:13:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:13:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1223 2024-09-22T00:13:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1225 2024-09-22T00:13:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1224 2024-09-22T00:13:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1222 2024-09-22T00:13:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:29.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:13:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:13:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:13:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:13:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:13:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:13:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:13:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:13:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:13:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1226 2024-09-22T00:13:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:13:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:13:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:13:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:13:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:13:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:13:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1227 2024-09-22T00:13:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1228 2024-09-22T00:13:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1229 2024-09-22T00:13:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:13:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:13:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:13:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:13:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:13:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:13:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:13:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:13:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:13:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:13:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:13:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:13:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1230 2024-09-22T00:13:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1231 2024-09-22T00:13:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:32.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:13:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:13:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:13:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1232 2024-09-22T00:13:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1235 2024-09-22T00:13:33.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1234 2024-09-22T00:13:34.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1233 2024-09-22T00:13:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:34.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:34.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:13:34.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:34.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:34.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:13:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:13:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:13:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:13:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:13:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:13:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:13:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:13:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1236 2024-09-22T00:13:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:13:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:13:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:13:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:13:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:13:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:13:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1237 2024-09-22T00:13:36.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1239 2024-09-22T00:13:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1238 2024-09-22T00:13:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:13:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:13:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:13:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:13:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:13:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:13:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:13:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:13:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:13:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:13:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:13:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:13:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1240 2024-09-22T00:13:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1241 2024-09-22T00:13:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1243 2024-09-22T00:13:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1242 2024-09-22T00:13:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:13:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:13:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:13:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1245 2024-09-22T00:13:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1244 2024-09-22T00:13:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:13:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:13:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:13:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:13:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:13:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:13:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:13:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:13:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:13:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1246 2024-09-22T00:13:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:13:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:13:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:13:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:13:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:13:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:13:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1248 2024-09-22T00:13:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1249 2024-09-22T00:13:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1247 2024-09-22T00:13:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:13:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:13:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:13:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:13:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:13:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:13:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1250 2024-09-22T00:13:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1251 2024-09-22T00:13:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:13:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:13:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:13:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:13:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:13:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:13:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1252 2024-09-22T00:13:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1253 2024-09-22T00:13:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:13:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:13:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:13:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:43.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:43.987 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:13:43.985+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3392ea0 2024-09-22T00:13:43.987 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:13:43.985+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3393110 2024-09-22T00:13:43.987 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:13:43.985+0000 7f54f2d95640 -1 reset not still connected to 0x5617f33931e0 2024-09-22T00:13:43.988 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:13:43.985+0000 7f54f2d95640 -1 reset not still connected to 0x5617f4b65520 2024-09-22T00:13:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1254 2024-09-22T00:13:43.992 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:13:43.985+0000 7f54f2d95640 -1 reset not still connected to 0x5617f711e9c0 2024-09-22T00:13:43.995 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:13:43.989+0000 7f54f2d95640 -1 reset not still connected to 0x5617ff240dd0 2024-09-22T00:13:44.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1255 2024-09-22T00:13:44.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:44.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:13:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:13:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:13:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:13:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:13:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:13:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:13:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:13:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:13:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:13:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:13:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:13:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:13:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:13:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:13:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1257 2024-09-22T00:13:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1256 2024-09-22T00:13:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1258 2024-09-22T00:13:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1259 2024-09-22T00:13:46.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:13:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:13:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:13:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:13:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:13:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:13:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1261 2024-09-22T00:13:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:13:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:13:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:13:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:13:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:13:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:13:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1260 2024-09-22T00:13:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:48.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1262 2024-09-22T00:13:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1263 2024-09-22T00:13:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1265 2024-09-22T00:13:49.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1264 2024-09-22T00:13:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:13:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:13:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:13:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:13:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:13:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:13:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:13:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:13:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:13:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:13:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:13:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:13:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:13:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:13:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:13:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1269 2024-09-22T00:13:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:13:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:13:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:13:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:51.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1267 2024-09-22T00:13:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1268 2024-09-22T00:13:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1266 2024-09-22T00:13:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:13:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:13:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:13:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:13:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:13:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:13:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:13:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:13:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:13:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1270 2024-09-22T00:13:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:13:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:13:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:13:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1271 2024-09-22T00:13:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1273 2024-09-22T00:13:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:13:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:13:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:13:53.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1275 2024-09-22T00:13:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1274 2024-09-22T00:13:54.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1272 2024-09-22T00:13:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:13:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:13:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:13:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:13:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:13:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:13:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:13:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:13:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:13:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1276 2024-09-22T00:13:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:13:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:13:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:13:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:13:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:13:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:13:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1277 2024-09-22T00:13:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1279 2024-09-22T00:13:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1278 2024-09-22T00:13:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:13:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:13:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:13:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:13:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:13:57.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:13:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:13:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:13:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:13:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:13:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:13:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:13:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1281 2024-09-22T00:13:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1280 2024-09-22T00:13:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:13:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:13:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:13:58.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1282 2024-09-22T00:13:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1285 2024-09-22T00:13:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1283 2024-09-22T00:13:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1284 2024-09-22T00:13:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:13:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:13:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:13:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:13:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:13:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:13:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:13:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:13:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:13:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:13:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:13:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:13:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:14:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1286 2024-09-22T00:14:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:14:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:14:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:14:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:14:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:14:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:14:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:14:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:14:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:14:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:14:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:14:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:14:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:14:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1288 2024-09-22T00:14:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1289 2024-09-22T00:14:01.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1287 2024-09-22T00:14:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:14:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:14:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:14:01.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:14:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:14:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:14:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:14:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:14:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:14:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:14:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:14:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:14:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:14:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:14:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:14:02.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1291 2024-09-22T00:14:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1290 2024-09-22T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:14:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:14:02.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:14:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1293 2024-09-22T00:14:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:14:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:14:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:14:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:14:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:14:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:14:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1292 2024-09-22T00:14:03.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:14:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:14:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1294 2024-09-22T00:14:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1295 2024-09-22T00:14:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:14:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:14:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:14:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:14:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:14:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:14:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:14:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:14:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:14:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 64582 2024-09-22T00:14:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:14:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:14:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:14:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:14:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:14:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:14:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1296 2024-09-22T00:14:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:14:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 64713 2024-09-22T00:14:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:14:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:14:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:14:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:14:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:14:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:14:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:14:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1298 2024-09-22T00:14:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:14:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:14:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:14:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:14:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:14:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:14:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1297 2024-09-22T00:14:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:14:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 64840 2024-09-22T00:14:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:14:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:14:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:14:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:14:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:14:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:14:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:14:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:14:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 64988 2024-09-22T00:14:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:14:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1299 2024-09-22T00:14:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:14:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1300 2024-09-22T00:14:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:14:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:14:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:14:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:14:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:14:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:14:08.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:14:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1301 2024-09-22T00:14:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:14:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:14:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:14:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:14:09.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:14:09.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:14:09.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:14:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:14:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 65155 2024-09-22T00:14:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1302 2024-09-22T00:14:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:14:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:14:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 65303 2024-09-22T00:14:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1303 2024-09-22T00:14:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:14:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 65431 2024-09-22T00:14:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:14:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:14:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 65599 2024-09-22T00:14:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:14:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 65766 2024-09-22T00:14:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:14:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:14:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 64460 2024-09-22T00:16:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:16:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:16:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:16:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 64583 2024-09-22T00:16:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:16:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:16:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:16:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 64714 2024-09-22T00:16:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:16:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:16:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:16:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 64841 2024-09-22T00:16:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:16:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:16:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:16:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 64989 2024-09-22T00:16:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:16:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:16:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:16:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 65156 2024-09-22T00:16:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:16:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:16:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:16:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 65304 2024-09-22T00:16:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:16:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:16:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:16:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 65432 2024-09-22T00:16:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:16:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:16:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:16:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 65600 2024-09-22T00:16:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:16:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:16:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:16:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 65767 2024-09-22T00:16:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:16:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:16:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:16:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:16:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:16:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:16:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 1 2024-09-22T00:16:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:16:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd0 2024-09-22T00:16:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:16:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary1 2024-09-22T00:16:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 2 2024-09-22T00:16:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary1 2024-09-22T00:16:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:16:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary1 2024-09-22T00:16:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd1 2024-09-22T00:16:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary2 2024-09-22T00:16:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary2 2024-09-22T00:16:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:16:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary2 2024-09-22T00:16:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:16:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 3 2024-09-22T00:16:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:16:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd2 2024-09-22T00:16:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary3 2024-09-22T00:16:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary3 2024-09-22T00:16:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:16:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary3 2024-09-22T00:16:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:16:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 4 2024-09-22T00:16:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:16:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd3 2024-09-22T00:16:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary4 2024-09-22T00:16:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary4 2024-09-22T00:16:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:16:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary4 2024-09-22T00:16:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:16:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 5 2024-09-22T00:16:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:16:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd4 2024-09-22T00:16:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary5 2024-09-22T00:16:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary5 2024-09-22T00:16:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:16:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary5 2024-09-22T00:16:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:16:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 6 2024-09-22T00:16:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:16:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd5 2024-09-22T00:16:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary6 2024-09-22T00:16:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary6 2024-09-22T00:16:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:16:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary6 2024-09-22T00:16:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:16:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:16:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:16:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:16:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 8 2024-09-22T00:16:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:16:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:16:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd7 2024-09-22T00:16:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary8 2024-09-22T00:16:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:16:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary8 2024-09-22T00:16:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:16:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 71925 2024-09-22T00:16:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:16:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary8 2024-09-22T00:16:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 10 2024-09-22T00:16:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd9 2024-09-22T00:16:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary10 2024-09-22T00:16:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary10 2024-09-22T00:16:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:16:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary10 2024-09-22T00:16:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 9 2024-09-22T00:16:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd8 2024-09-22T00:16:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary9 2024-09-22T00:16:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary9 2024-09-22T00:16:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:16:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary9 2024-09-22T00:16:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 7 2024-09-22T00:16:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd6 2024-09-22T00:16:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary7 2024-09-22T00:16:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary7 2024-09-22T00:16:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:16:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary7 2024-09-22T00:16:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-22T00:16:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:16:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-22T00:16:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:16:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-22T00:16:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:16:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-22T00:16:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:16:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-22T00:16:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:16:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-22T00:16:46.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:16:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-22T00:16:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:16:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-22T00:16:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:16:46.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-22T00:16:46.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-22T00:16:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:16:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:17:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=116015ade96925eca100a2c143bc6678 2024-09-22T00:17:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-22T00:17:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-22T00:17:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:17:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-22T00:17:18.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=47d92a3b9a2b6d9a5efbfa798a28b602 2024-09-22T00:17:18.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-22T00:17:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-22T00:17:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:17:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-22T00:17:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=4207f404c768a4eccd68e9a24bb00405 2024-09-22T00:17:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-22T00:17:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-22T00:17:18.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:18.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:18.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:17:18.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-22T00:17:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=33f6f09a6ff39f78398b236d66a0c4f8 2024-09-22T00:17:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-22T00:17:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-22T00:17:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:17:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-22T00:17:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=b110a52749015587bdfe3eb5727f4bfb 2024-09-22T00:17:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-22T00:17:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-22T00:17:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:17:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-22T00:17:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:17:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-22T00:17:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:17:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=fbf058bf646280ed0db7f4ccc7798fb9 2024-09-22T00:17:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-22T00:17:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=5db53672605cfa9f271c19edfabc9e44 2024-09-22T00:17:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-22T00:17:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=768844f7610f9933409c9aae89238e07 2024-09-22T00:17:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-22T00:17:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-22T00:17:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:17:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-22T00:17:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-22T00:17:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:17:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-22T00:17:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=e9cae487043a34bd01a876f3cfe42b65 2024-09-22T00:17:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-22T00:17:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-22T00:17:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:17:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-22T00:17:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-22T00:17:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:17:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-22T00:17:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=60f2f6bed4ae58aaa7d7a78a423878a1 2024-09-22T00:17:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-22T00:17:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-22T00:17:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:17:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-22T00:17:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:17:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-22T00:17:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:17:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:20.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:17:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:17:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:17:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-22T00:17:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:17:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-22T00:17:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:17:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-22T00:17:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:17:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:17:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9cf9377c-212a-418d-80e7-6b582738ce1e 2024-09-22T00:17:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20 2024-09-22T00:17:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:17:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20' 2024-09-22T00:17:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:20.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:20.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:17:20.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-22 00:17:20' 2024-09-22T00:17:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:17:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:20' 2024-09-22T00:17:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3629056.0,"last_snapshot_bytes":1654784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964039,"remote_snapshot_timestamp":1726964039,"replay_state":"idle"}2024-09-22 00:17:05image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":884736.0,"last_snapshot_bytes":1560576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964050,"remote_snapshot_timestamp":1726964050,"replay_state":"idle"}2024-09-22 00:17:05image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1705984.0,"last_snapshot_bytes":1691648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964041,"remote_snapshot_timestamp":1726964041,"replay_state":"idle"}2024-09-22 00:17:05image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":944128.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964042,"remote_snapshot_timestamp":1726964042,"replay_state":"idle"}2024-09-22 00:17:05image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1726464.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964042,"remote_snapshot_timestamp":1726964042,"replay_state":"idle"}2024-09-22 00:17:05image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1720320.0,"last_snapshot_bytes":1671168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964046,"remote_snapshot_timestamp":1726964046,"replay_state":"idle"}2024-09-22 00:17:05image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":143360.0,"last_snapshot_bytes":131072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964047,"remote_snapshot_timestamp":1726964047,"replay_state":"idle"}2024-09-22 00:17:05image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1699840.0,"last_snapshot_bytes":1658880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964048,"remote_snapshot_timestamp":1726964048,"replay_state":"idle"}2024-09-22 00:17:05image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2750464.0,"last_snapshot_bytes":118784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964048,"remote_snapshot_timestamp":1726964048,"replay_state":"idle"}2024-09-22 00:17:05' 2024-09-22T00:17:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:17:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:17:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:20.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:20.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-22T00:17:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:17:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:17:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:17:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:17:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-22T00:17:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:17:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-22T00:17:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:17:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:17:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:17:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-22T00:17:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:17:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:17:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-22T00:17:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:17:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:17:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-22T00:17:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:17:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:17:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:17:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-22T00:17:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:17:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-22T00:17:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:17:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-22T00:17:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:17:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:17:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 85b93c29-7d43-4d0a-8feb-205586f1412b 2024-09-22T00:17:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20 2024-09-22T00:17:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:17:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:17:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:21' 2024-09-22T00:17:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:17:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:17:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 57679ffa-22a0-4454-9dd1-24f459258a36 2024-09-22T00:17:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20 2024-09-22T00:17:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:17:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:17:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:21' 2024-09-22T00:17:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:17:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: b40c8172-0f71-438a-aaba-7aeb6f08c4a1 2024-09-22T00:17:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20 2024-09-22T00:17:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:17:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:17:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:21' 2024-09-22T00:17:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-22 00:17:20' 2024-09-22T00:17:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:17:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:20' 2024-09-22T00:17:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:17:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:17:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-22T00:17:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:17:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:21.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:21.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-22 00:17:20' 2024-09-22T00:17:21.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:21.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:21.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:17:21.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:21.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:20' 2024-09-22T00:17:21.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-22 00:17:20' 2024-09-22T00:17:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:17:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:20' 2024-09-22T00:17:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-22T00:17:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:17:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-22T00:17:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3629056.0,"last_snapshot_bytes":1654784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964039,"remote_snapshot_timestamp":1726964039,"replay_state":"idle"}2024-09-22 00:17:05image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1705984.0,"last_snapshot_bytes":1691648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964041,"remote_snapshot_timestamp":1726964041,"replay_state":"idle"}2024-09-22 00:17:05image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":944128.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964042,"remote_snapshot_timestamp":1726964042,"replay_state":"idle"}2024-09-22 00:17:05image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1720320.0,"last_snapshot_bytes":1671168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964046,"remote_snapshot_timestamp":1726964046,"replay_state":"idle"}2024-09-22 00:17:05image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":143360.0,"last_snapshot_bytes":131072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964047,"remote_snapshot_timestamp":1726964047,"replay_state":"idle"}2024-09-22 00:17:05image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2750464.0,"last_snapshot_bytes":118784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964048,"remote_snapshot_timestamp":1726964048,"replay_state":"idle"}2024-09-22 00:17:05' 2024-09-22T00:17:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:17:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:17:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-22T00:17:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:17:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:17:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9cf9377c-212a-418d-80e7-6b582738ce1e 2024-09-22T00:17:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20 2024-09-22T00:17:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:17:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20' 2024-09-22T00:17:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:17:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-22 00:17:20' 2024-09-22T00:17:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:17:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:20' 2024-09-22T00:17:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3629056.0,"last_snapshot_bytes":1654784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964039,"remote_snapshot_timestamp":1726964039,"replay_state":"idle"}2024-09-22 00:17:05image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1705984.0,"last_snapshot_bytes":1691648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964041,"remote_snapshot_timestamp":1726964041,"replay_state":"idle"}2024-09-22 00:17:05image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":944128.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964042,"remote_snapshot_timestamp":1726964042,"replay_state":"idle"}2024-09-22 00:17:05image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1720320.0,"last_snapshot_bytes":1671168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964046,"remote_snapshot_timestamp":1726964046,"replay_state":"idle"}2024-09-22 00:17:05image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":143360.0,"last_snapshot_bytes":131072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964047,"remote_snapshot_timestamp":1726964047,"replay_state":"idle"}2024-09-22 00:17:05image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2750464.0,"last_snapshot_bytes":118784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964048,"remote_snapshot_timestamp":1726964048,"replay_state":"idle"}2024-09-22 00:17:05' 2024-09-22T00:17:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:17:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:21.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:17:21.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3629056.0,"last_snapshot_bytes":1654784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964039,"remote_snapshot_timestamp":1726964039,"replay_state":"idle"}2024-09-22 00:17:05image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1705984.0,"last_snapshot_bytes":1691648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964041,"remote_snapshot_timestamp":1726964041,"replay_state":"idle"}2024-09-22 00:17:05image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":944128.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964042,"remote_snapshot_timestamp":1726964042,"replay_state":"idle"}2024-09-22 00:17:05image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1720320.0,"last_snapshot_bytes":1671168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964046,"remote_snapshot_timestamp":1726964046,"replay_state":"idle"}2024-09-22 00:17:05image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":143360.0,"last_snapshot_bytes":131072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964047,"remote_snapshot_timestamp":1726964047,"replay_state":"idle"}2024-09-22 00:17:05image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2750464.0,"last_snapshot_bytes":118784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964048,"remote_snapshot_timestamp":1726964048,"replay_state":"idle"}2024-09-22 00:17:05' 2024-09-22T00:17:21.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:21.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:17:21.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:21.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:17:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-22T00:17:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:17:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-22T00:17:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3629056.0,"last_snapshot_bytes":1654784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964039,"remote_snapshot_timestamp":1726964039,"replay_state":"idle"}42634269mirror.0smithi0602024-09-22 00:17:05cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1705984.0,"last_snapshot_bytes":1691648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964041,"remote_snapshot_timestamp":1726964041,"replay_state":"idle"}42634269mirror.0smithi0602024-09-22 00:17:05cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":944128.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964042,"remote_snapshot_timestamp":1726964042,"replay_state":"idle"}42634269mirror.0smithi0602024-09-22 00:17:05cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:17:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1720320.0,"last_snapshot_bytes":1671168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964046,"remote_snapshot_timestamp":1726964046,"replay_state":"idle"}42634269mirror.0smithi0602024-09-22 00:17:05cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+stopping_replayremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20' 2024-09-22T00:17:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:17:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:17:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-22T00:17:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:17:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:17:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-22T00:17:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-22T00:17:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:17:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:17:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-22T00:17:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:17:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:17:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-22T00:17:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:17:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:17:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:17:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f3d555de-4bdb-4256-96ae-c858dd8364e9 2024-09-22T00:17:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20 2024-09-22T00:17:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:17:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:17:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:22' 2024-09-22T00:17:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:17:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:22.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:22.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-22 00:17:20' 2024-09-22T00:17:22.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:17:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:20' 2024-09-22T00:17:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-22T00:17:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:17:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:17:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:17:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: d3b5b5e9-88cd-4138-ba00-75ea869fb1dd 2024-09-22T00:17:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20 2024-09-22T00:17:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:17:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:17:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:22' 2024-09-22T00:17:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:17:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-22 00:17:20' 2024-09-22T00:17:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:17:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:20' 2024-09-22T00:17:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-22T00:17:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:17:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:17:22.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-22T00:17:22.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:22.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:17:22.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:22.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:22.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:22.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:22.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:17:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:17:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f74c1283-be68-450d-aa81-c2905ec3b6bd 2024-09-22T00:17:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20 2024-09-22T00:17:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:17:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:17:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":944128.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964042,"remote_snapshot_timestamp":1726964042,"replay_state":"idle"} 2024-09-22T00:17:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:05' 2024-09-22T00:17:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:17:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-22 00:17:20' 2024-09-22T00:17:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:17:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:20' 2024-09-22T00:17:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":944128.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964042,"remote_snapshot_timestamp":1726964042,"replay_state":"idle"}2024-09-22 00:17:05image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22' 2024-09-22T00:17:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:17:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:17:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-22T00:17:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:17:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:17:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 46a0a802-96d3-4bb0-bdb7-f572fb64611f 2024-09-22T00:17:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20 2024-09-22T00:17:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:17:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:17:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:22' 2024-09-22T00:17:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:17:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:17:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f038f20d-309b-496d-b976-53bbf0460858 2024-09-22T00:17:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20 2024-09-22T00:17:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:17:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:17:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:22' 2024-09-22T00:17:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-22 00:17:20' 2024-09-22T00:17:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:17:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:20' 2024-09-22T00:17:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:17:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-22 00:17:20' 2024-09-22T00:17:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:17:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:20' 2024-09-22T00:17:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":944128.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964042,"remote_snapshot_timestamp":1726964042,"replay_state":"idle"}2024-09-22 00:17:05image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22' 2024-09-22T00:17:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:17:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:17:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-22T00:17:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:17:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":944128.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964042,"remote_snapshot_timestamp":1726964042,"replay_state":"idle"}2024-09-22 00:17:05image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22' 2024-09-22T00:17:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:17:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:17:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":944128.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964042,"remote_snapshot_timestamp":1726964042,"replay_state":"idle"}2024-09-22 00:17:05image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22' 2024-09-22T00:17:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:17:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:17:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-22T00:17:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:17:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-22T00:17:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:17:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-22T00:17:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:17:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-22T00:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-22T00:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-22T00:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-22T00:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:17:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-22T00:17:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:17:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 85b93c29-7d43-4d0a-8feb-205586f1412b 2024-09-22T00:17:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:17:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:21 2024-09-22T00:17:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:17:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20' 2024-09-22T00:17:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:17:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:21' 2024-09-22T00:17:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-22 00:17:21' 2024-09-22T00:17:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:17:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:21' 2024-09-22T00:17:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":944128.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964042,"remote_snapshot_timestamp":1726964042,"replay_state":"idle"}2024-09-22 00:17:05image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22' 2024-09-22T00:17:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:17:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:17:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:17:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 1efac4d0-57a5-4f9c-ad45-0595686a4e3b 2024-09-22T00:17:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20 2024-09-22T00:17:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:17:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:17:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:22' 2024-09-22T00:17:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:17:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-22T00:17:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:17:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:22.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-22 00:17:20' 2024-09-22T00:17:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:17:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:20' 2024-09-22T00:17:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:17:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: b40c8172-0f71-438a-aaba-7aeb6f08c4a1 2024-09-22T00:17:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:17:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:21 2024-09-22T00:17:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:17:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20' 2024-09-22T00:17:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:17:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 57679ffa-22a0-4454-9dd1-24f459258a36 2024-09-22T00:17:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:17:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:21 2024-09-22T00:17:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:17:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20' 2024-09-22T00:17:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:17:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:17:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:21' 2024-09-22T00:17:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-22 00:17:21' 2024-09-22T00:17:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:17:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:21' 2024-09-22T00:17:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:21' 2024-09-22T00:17:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-22 00:17:21' 2024-09-22T00:17:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:17:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:21' 2024-09-22T00:17:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":944128.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964042,"remote_snapshot_timestamp":1726964042,"replay_state":"idle"}42634269mirror.0smithi0602024-09-22 00:17:05cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:17:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20' 2024-09-22T00:17:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:17:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:21' 2024-09-22T00:17:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:17:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:23.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-22T00:17:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:17:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:17:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-22T00:17:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":944128.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964042,"remote_snapshot_timestamp":1726964042,"replay_state":"idle"}2024-09-22 00:17:05image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:21image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:17:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:17:22' 2024-09-22T00:17:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:17:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:20' 2024-09-22T00:17:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:17:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-22T00:17:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:17:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:17:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:17:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+stopping_replayremote image demoted42634269mirror.0smithi0602024-09-22 00:17:23cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:17:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20' 2024-09-22T00:17:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:17:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:21' 2024-09-22T00:17:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+stopping_replayremote image demoted42634269mirror.0smithi0602024-09-22 00:17:23cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:17:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20' 2024-09-22T00:17:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:17:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:17:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:21' 2024-09-22T00:17:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:17:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-22T00:17:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:17:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:17:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-22T00:17:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:17:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:17:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-22T00:17:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-22T00:17:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:17:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:17:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-22T00:17:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd0 2024-09-22T00:17:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-22T00:17:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:17:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-22T00:17:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:17:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-22T00:17:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-22T00:17:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:17:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-22T00:17:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:17:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f3d555de-4bdb-4256-96ae-c858dd8364e9 2024-09-22T00:17:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:17:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:22 2024-09-22T00:17:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:17:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20' 2024-09-22T00:17:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:17:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:22' 2024-09-22T00:17:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-22 00:17:22' 2024-09-22T00:17:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:17:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:22' 2024-09-22T00:17:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:17:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: d3b5b5e9-88cd-4138-ba00-75ea869fb1dd 2024-09-22T00:17:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:17:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:22 2024-09-22T00:17:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:17:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20' 2024-09-22T00:17:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:17:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:22' 2024-09-22T00:17:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-22 00:17:22' 2024-09-22T00:17:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:17:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:22' 2024-09-22T00:17:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-22T00:17:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:17:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-22T00:17:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-22T00:17:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:17:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-22T00:17:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:23cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:17:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20' 2024-09-22T00:17:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:17:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:22' 2024-09-22T00:17:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:17:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-22T00:17:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:17:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:17:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-22T00:17:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:17:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 46a0a802-96d3-4bb0-bdb7-f572fb64611f 2024-09-22T00:17:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:17:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:22 2024-09-22T00:17:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:17:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20' 2024-09-22T00:17:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:17:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:23cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:17:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20' 2024-09-22T00:17:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:22' 2024-09-22T00:17:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-22 00:17:22' 2024-09-22T00:17:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:17:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:17:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:22' 2024-09-22T00:17:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:22' 2024-09-22T00:17:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:17:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-22T00:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:17:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-22T00:17:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:17:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f038f20d-309b-496d-b976-53bbf0460858 2024-09-22T00:17:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:17:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:22 2024-09-22T00:17:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:17:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20' 2024-09-22T00:17:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:17:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:22' 2024-09-22T00:17:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-22 00:17:22' 2024-09-22T00:17:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:17:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:22' 2024-09-22T00:17:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-22T00:17:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:17:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-22T00:17:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:23cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:17:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20' 2024-09-22T00:17:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f74c1283-be68-450d-aa81-c2905ec3b6bd 2024-09-22T00:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:23 2024-09-22T00:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20' 2024-09-22T00:17:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:17:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:17:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:22' 2024-09-22T00:17:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:17:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:23' 2024-09-22T00:17:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-22 00:17:23' 2024-09-22T00:17:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:17:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:23' 2024-09-22T00:17:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-22T00:17:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:17:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:17:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-22T00:17:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:17:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:17:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:17:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:17:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-22T00:17:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:23cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:17:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20' 2024-09-22T00:17:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:17:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:17:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:17:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-22T00:17:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-22T00:17:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-22T00:17:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:22' 2024-09-22T00:17:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:17:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-22T00:17:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:17:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:17:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-22T00:17:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:17:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:17:24.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:17:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:17:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:17:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-22T00:17:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:17:24.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:17:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:17:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:17:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-22T00:17:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-22T00:17:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:17:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 1efac4d0-57a5-4f9c-ad45-0595686a4e3b 2024-09-22T00:17:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:17:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:17:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:22 2024-09-22T00:17:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:17:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:17:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:17:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:17:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:20' 2024-09-22T00:17:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:17:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:17:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:17:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:17:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:17:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:17:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd1 2024-09-22T00:17:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:17:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-22T00:17:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:23cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:17:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20' 2024-09-22T00:17:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:17:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:17:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:17:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:17:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:17:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:22' 2024-09-22T00:17:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-22 00:17:22' 2024-09-22T00:17:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:17:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:17:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:22' 2024-09-22T00:17:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:17:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:17:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:17:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:23' 2024-09-22T00:17:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:17:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:24.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-22T00:17:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:17:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:17:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-22T00:17:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd2 2024-09-22T00:17:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-22T00:17:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:17:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd4 2024-09-22T00:17:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-22T00:17:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd5 2024-09-22T00:17:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd3 2024-09-22T00:17:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-22T00:17:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:17:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-22T00:17:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:17:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:17:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:23cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:17:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary7f038f20d-309b-496d-b976-53bbf0460858up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:21cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:17:22cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:17:20' 2024-09-22T00:17:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:17:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:17:24.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:22' 2024-09-22T00:17:24.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:17:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:17:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:17:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:17:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:17:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:17:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-22T00:17:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:17:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:17:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:17:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-22T00:17:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:17:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:17:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:17:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:17:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:17:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:17:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:17:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:17:25.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:17:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:17:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-22T00:17:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:17:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:17:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-22T00:17:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-22T00:17:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-22T00:17:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd7 2024-09-22T00:17:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-22T00:17:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd6 2024-09-22T00:17:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd8 2024-09-22T00:17:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:17:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:17:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-22T00:17:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-22T00:17:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:17:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd9 2024-09-22T00:17:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:17:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-22T00:17:26.894 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:17:26.892+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7d9d0dad0 2024-09-22T00:17:56.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=116015ade96925eca100a2c143bc6678 2024-09-22T00:17:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-22T00:17:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 116015ade96925eca100a2c143bc6678 != \1\1\6\0\1\5\a\d\e\9\6\9\2\5\e\c\a\1\0\0\a\2\c\1\4\3\b\c\6\6\7\8 ]] 2024-09-22T00:17:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=768844f7610f9933409c9aae89238e07 2024-09-22T00:17:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-22T00:17:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=60f2f6bed4ae58aaa7d7a78a423878a1 2024-09-22T00:17:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-22T00:17:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 768844f7610f9933409c9aae89238e07 != \7\6\8\8\4\4\f\7\6\1\0\f\9\9\3\3\4\0\9\c\9\a\a\e\8\9\2\3\8\e\0\7 ]] 2024-09-22T00:17:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 60f2f6bed4ae58aaa7d7a78a423878a1 != \6\0\f\2\f\6\b\e\d\4\a\e\5\8\a\a\a\7\d\7\a\7\8\a\4\2\3\8\7\8\a\1 ]] 2024-09-22T00:17:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=47d92a3b9a2b6d9a5efbfa798a28b602 2024-09-22T00:17:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-22T00:17:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 47d92a3b9a2b6d9a5efbfa798a28b602 != \4\7\d\9\2\a\3\b\9\a\2\b\6\d\9\a\5\e\f\b\f\a\7\9\8\a\2\8\b\6\0\2 ]] 2024-09-22T00:17:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=33f6f09a6ff39f78398b236d66a0c4f8 2024-09-22T00:17:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-22T00:17:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 33f6f09a6ff39f78398b236d66a0c4f8 != \3\3\f\6\f\0\9\a\6\f\f\3\9\f\7\8\3\9\8\b\2\3\6\d\6\6\a\0\c\4\f\8 ]] 2024-09-22T00:17:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=b110a52749015587bdfe3eb5727f4bfb 2024-09-22T00:17:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-22T00:17:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=4207f404c768a4eccd68e9a24bb00405 2024-09-22T00:17:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-22T00:17:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ b110a52749015587bdfe3eb5727f4bfb != \b\1\1\0\a\5\2\7\4\9\0\1\5\5\8\7\b\d\f\e\3\e\b\5\7\2\7\f\4\b\f\b ]] 2024-09-22T00:17:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:17:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 71926 2024-09-22T00:17:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=e9cae487043a34bd01a876f3cfe42b65 2024-09-22T00:17:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-22T00:17:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 4207f404c768a4eccd68e9a24bb00405 != \4\2\0\7\f\4\0\4\c\7\6\8\a\4\e\c\c\d\6\8\e\9\a\2\4\b\b\0\0\4\0\5 ]] 2024-09-22T00:17:59.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=fbf058bf646280ed0db7f4ccc7798fb9 2024-09-22T00:17:59.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-22T00:17:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ e9cae487043a34bd01a876f3cfe42b65 != \e\9\c\a\e\4\8\7\0\4\3\a\3\4\b\d\0\1\a\8\7\6\f\3\c\f\e\4\2\b\6\5 ]] 2024-09-22T00:17:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ fbf058bf646280ed0db7f4ccc7798fb9 != \f\b\f\0\5\8\b\f\6\4\6\2\8\0\e\d\0\d\b\7\f\4\c\c\c\7\7\9\8\f\b\9 ]] 2024-09-22T00:17:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=5db53672605cfa9f271c19edfabc9e44 2024-09-22T00:17:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-22T00:17:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 5db53672605cfa9f271c19edfabc9e44 != \5\d\b\5\3\6\7\2\6\0\5\c\f\a\9\f\2\7\1\c\1\9\e\d\f\a\b\c\9\e\4\4 ]] 2024-09-22T00:17:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:17:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 71927 2024-09-22T00:17:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:17:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 71930 2024-09-22T00:17:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:17:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 71932 2024-09-22T00:17:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:17:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 71934 2024-09-22T00:17:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:17:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 71936 2024-09-22T00:17:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:17:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 71938 2024-09-22T00:17:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:17:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 71939 2024-09-22T00:17:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:17:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 71940 2024-09-22T00:17:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:17:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:17:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-22T00:17:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-22T00:17:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:17:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:17:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-22T00:17:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:17:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:17:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:17:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:17:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:17:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:17:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:17:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:18:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-22T00:18:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:18:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:18:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:18:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 1efac4d0-57a5-4f9c-ad45-0595686a4e3b 2024-09-22T00:18:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:18:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63647744.0,"last_snapshot_bytes":63647744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"} 2024-09-22T00:18:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:18:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:35 2024-09-22T00:18:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:18:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:18:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:18:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:18:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:18:00' 2024-09-22T00:18:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:18:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:18:00.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:18:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63647744.0,"last_snapshot_bytes":63647744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"} 2024-09-22T00:18:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:18:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:18:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:18:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:18:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:18:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:18:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:35' 2024-09-22T00:18:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-22 00:17:35' 2024-09-22T00:18:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:18:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:35' 2024-09-22T00:18:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:18:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary11efac4d0-57a5-4f9c-ad45-0595686a4e3bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63647744.0,"last_snapshot_bytes":63647744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00image-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+replayingreplaying, {"bytes_per_second":36128768.0,"bytes_per_snapshot":72257536.0,"last_snapshot_bytes":72257536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+replayingreplaying, {"bytes_per_second":36159488.0,"bytes_per_snapshot":72318976.0,"last_snapshot_bytes":72318976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+replayingreplaying, {"bytes_per_second":36175872.0,"bytes_per_snapshot":72351744.0,"last_snapshot_bytes":72351744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72302592.0,"last_snapshot_bytes":72302592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63184896.0,"last_snapshot_bytes":63184896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964243,"remote_snapshot_timestamp":1726964243,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00image-primary7f038f20d-309b-496d-b976-53bbf0460858up+replayingreplaying, {"bytes_per_second":6840320.0,"bytes_per_snapshot":66830336.0,"last_snapshot_bytes":66830336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+replayingreplaying, {"bytes_per_second":35872768.0,"bytes_per_snapshot":71745536.0,"last_snapshot_bytes":71745536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64917504.0,"last_snapshot_bytes":64917504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00' 2024-09-22T00:18:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:18:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:18:01.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:35' 2024-09-22T00:18:01.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:18:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:18:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:18:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-22T00:18:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:18:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:18:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-22T00:18:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-22T00:18:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:18:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-22T00:18:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:18:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:18:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary1 2024-09-22T00:18:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary1 2024-09-22T00:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-22T00:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-22T00:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-22T00:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:18:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:18:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-22T00:18:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:18:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:18:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:18:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 46a0a802-96d3-4bb0-bdb7-f572fb64611f 2024-09-22T00:18:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:18:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":36159488.0,"bytes_per_snapshot":72318976.0,"last_snapshot_bytes":72318976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"} 2024-09-22T00:18:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:18:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:17:35 2024-09-22T00:18:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:18:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:18:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:18:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:18:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:18:00' 2024-09-22T00:18:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:18:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:18:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:18:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":36159488.0,"bytes_per_snapshot":72318976.0,"last_snapshot_bytes":72318976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"} 2024-09-22T00:18:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:18:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:18:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:18:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:18:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:18:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:18:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:35' 2024-09-22T00:18:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-22 00:17:35' 2024-09-22T00:18:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:18:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:17:35' 2024-09-22T00:18:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:18:04.662 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:18:04.655+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7c58148f0 2024-09-22T00:18:04.662 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:18:04.655+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7c5814d00 2024-09-22T00:18:04.662 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:18:04.655+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7c9c52000 2024-09-22T00:18:04.662 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:18:04.659+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7d8c545b0 2024-09-22T00:18:04.662 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:18:04.659+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7d8c54d00 2024-09-22T00:18:04.662 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:18:04.659+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7d9d0dad0 2024-09-22T00:18:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK942614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+replayingreplaying, {"bytes_per_second":36128768.0,"bytes_per_snapshot":72257536.0,"last_snapshot_bytes":72257536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00image-primary246a0a802-96d3-4bb0-bdb7-f572fb64611fup+replayingreplaying, {"bytes_per_second":36159488.0,"bytes_per_snapshot":72318976.0,"last_snapshot_bytes":72318976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+replayingreplaying, {"bytes_per_second":36175872.0,"bytes_per_snapshot":72351744.0,"last_snapshot_bytes":72351744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72302592.0,"last_snapshot_bytes":72302592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63184896.0,"last_snapshot_bytes":63184896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964243,"remote_snapshot_timestamp":1726964243,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00image-primary7f038f20d-309b-496d-b976-53bbf0460858up+replayingreplaying, {"bytes_per_second":6840320.0,"bytes_per_snapshot":66830336.0,"last_snapshot_bytes":66830336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+replayingreplaying, {"bytes_per_second":35872768.0,"bytes_per_snapshot":71745536.0,"last_snapshot_bytes":71745536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64917504.0,"last_snapshot_bytes":64917504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:17:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:00' 2024-09-22T00:18:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:18:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:18:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:17:35' 2024-09-22T00:18:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:18:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:18:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:18:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-22T00:18:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:18:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:04.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:18:04.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-22T00:18:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-22T00:18:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:18:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-22T00:18:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:18:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:18:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary2 2024-09-22T00:18:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary2 2024-09-22T00:18:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-22T00:18:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-22T00:18:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:18:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:18:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-22T00:18:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:18:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:18:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-22T00:18:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:18:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:18:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:18:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f74c1283-be68-450d-aa81-c2905ec3b6bd 2024-09-22T00:18:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:18:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2411724.8,"bytes_per_snapshot":72351744.0,"last_snapshot_bytes":72351744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"} 2024-09-22T00:18:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:18:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:18:05 2024-09-22T00:18:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:18:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:18:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:18:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:18:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:18:05' 2024-09-22T00:18:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:18:07.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:18:07.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:18:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2411724.8,"bytes_per_snapshot":72351744.0,"last_snapshot_bytes":72351744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"} 2024-09-22T00:18:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:18:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:18:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:18:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:18:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:18:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:18:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:18:05' 2024-09-22T00:18:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-22 00:18:05' 2024-09-22T00:18:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:18:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:18:05' 2024-09-22T00:18:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:18:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK842614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+replayingreplaying, {"bytes_per_second":2408584.53,"bytes_per_snapshot":72257536.0,"last_snapshot_bytes":72257536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:05image-primary3f74c1283-be68-450d-aa81-c2905ec3b6bdup+replayingreplaying, {"bytes_per_second":2411724.8,"bytes_per_snapshot":72351744.0,"last_snapshot_bytes":72351744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:05image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+replayingreplaying, {"bytes_per_second":2410086.4,"bytes_per_snapshot":72302592.0,"last_snapshot_bytes":72302592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:05image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+replayingreplaying, {"bytes_per_second":2106163.2,"bytes_per_snapshot":63184896.0,"last_snapshot_bytes":63184896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964243,"remote_snapshot_timestamp":1726964243,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:05image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+replayingreplaying, {"bytes_per_second":2393292.8,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:05image-primary7f038f20d-309b-496d-b976-53bbf0460858up+replayingreplaying, {"bytes_per_second":2227677.87,"bytes_per_snapshot":66830336.0,"last_snapshot_bytes":66830336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:05image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+replayingreplaying, {"bytes_per_second":2391517.87,"bytes_per_snapshot":71745536.0,"last_snapshot_bytes":71745536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:05image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+replayingreplaying, {"bytes_per_second":2163916.8,"bytes_per_snapshot":64917504.0,"last_snapshot_bytes":64917504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:05' 2024-09-22T00:18:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:18:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:18:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:18:05' 2024-09-22T00:18:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:18:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:18:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:18:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-22T00:18:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:18:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:18:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-22T00:18:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-22T00:18:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:18:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-22T00:18:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:18:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:18:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary3 2024-09-22T00:18:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary3 2024-09-22T00:18:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-22T00:18:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-22T00:18:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:18:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:18:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-22T00:18:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:18:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:18:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-22T00:18:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:18:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:18:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:18:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 57679ffa-22a0-4454-9dd1-24f459258a36 2024-09-22T00:18:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:18:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2410086.4,"bytes_per_snapshot":72302592.0,"last_snapshot_bytes":72302592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"} 2024-09-22T00:18:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:18:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:18:05 2024-09-22T00:18:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:18:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:18:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:18:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:18:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:18:05' 2024-09-22T00:18:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:18:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:18:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:18:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2410086.4,"bytes_per_snapshot":72302592.0,"last_snapshot_bytes":72302592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"} 2024-09-22T00:18:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:18:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:18:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:18:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:18:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:18:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:18:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:18:05' 2024-09-22T00:18:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-22 00:18:05' 2024-09-22T00:18:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:18:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:18:05' 2024-09-22T00:18:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:18:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK742614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+replayingreplaying, {"bytes_per_second":2408584.53,"bytes_per_snapshot":72257536.0,"last_snapshot_bytes":72257536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:05image-primary457679ffa-22a0-4454-9dd1-24f459258a36up+replayingreplaying, {"bytes_per_second":2410086.4,"bytes_per_snapshot":72302592.0,"last_snapshot_bytes":72302592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:05image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+replayingreplaying, {"bytes_per_second":2106163.2,"bytes_per_snapshot":63184896.0,"last_snapshot_bytes":63184896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964243,"remote_snapshot_timestamp":1726964243,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:05image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+replayingreplaying, {"bytes_per_second":2393292.8,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:05image-primary7f038f20d-309b-496d-b976-53bbf0460858up+replayingreplaying, {"bytes_per_second":2227677.87,"bytes_per_snapshot":66830336.0,"last_snapshot_bytes":66830336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:05image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+replayingreplaying, {"bytes_per_second":2391517.87,"bytes_per_snapshot":71745536.0,"last_snapshot_bytes":71745536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:05image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+replayingreplaying, {"bytes_per_second":2163916.8,"bytes_per_snapshot":64917504.0,"last_snapshot_bytes":64917504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:10' 2024-09-22T00:18:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:18:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:18:10.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:18:05' 2024-09-22T00:18:10.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:18:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:18:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:18:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-22T00:18:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:18:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:18:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-22T00:18:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-22T00:18:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:18:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-22T00:18:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:18:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:18:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary4 2024-09-22T00:18:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary4 2024-09-22T00:18:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-22T00:18:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-22T00:18:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:18:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:18:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-22T00:18:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:18:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:18:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-22T00:18:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:18:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:18:13.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:18:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9cf9377c-212a-418d-80e7-6b582738ce1e 2024-09-22T00:18:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:18:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2106163.2,"bytes_per_snapshot":63184896.0,"last_snapshot_bytes":63184896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964243,"remote_snapshot_timestamp":1726964243,"replay_state":"idle"} 2024-09-22T00:18:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:18:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:18:05 2024-09-22T00:18:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:18:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:18:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:18:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:18:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:18:10' 2024-09-22T00:18:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:18:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:18:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:18:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2106163.2,"bytes_per_snapshot":63184896.0,"last_snapshot_bytes":63184896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964243,"remote_snapshot_timestamp":1726964243,"replay_state":"idle"} 2024-09-22T00:18:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:18:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:18:13.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:13.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:18:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:18:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:18:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:18:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:18:05' 2024-09-22T00:18:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-22 00:18:05' 2024-09-22T00:18:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:18:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:18:05' 2024-09-22T00:18:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:18:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK642614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+replayingreplaying, {"bytes_per_second":2408584.53,"bytes_per_snapshot":72257536.0,"last_snapshot_bytes":72257536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:10image-primary59cf9377c-212a-418d-80e7-6b582738ce1eup+replayingreplaying, {"bytes_per_second":2106163.2,"bytes_per_snapshot":63184896.0,"last_snapshot_bytes":63184896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964243,"remote_snapshot_timestamp":1726964243,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:10image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+replayingreplaying, {"bytes_per_second":2393292.8,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:10image-primary7f038f20d-309b-496d-b976-53bbf0460858up+replayingreplaying, {"bytes_per_second":2227677.87,"bytes_per_snapshot":66830336.0,"last_snapshot_bytes":66830336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:10image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+replayingreplaying, {"bytes_per_second":2391517.87,"bytes_per_snapshot":71745536.0,"last_snapshot_bytes":71745536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:10image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+replayingreplaying, {"bytes_per_second":2163916.8,"bytes_per_snapshot":64917504.0,"last_snapshot_bytes":64917504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:10' 2024-09-22T00:18:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:18:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:18:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:18:05' 2024-09-22T00:18:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:18:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:18:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:18:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-22T00:18:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:18:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:18:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-22T00:18:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-22T00:18:14.442 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:18:14.439+0000 7fab5fe19640 -1 reset not still connected to 0x5623f97bd860 2024-09-22T00:18:14.442 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:18:14.439+0000 7fab5fe19640 -1 reset not still connected to 0x5623f97bdba0 2024-09-22T00:18:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:18:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-22T00:18:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:18:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:18:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary5 2024-09-22T00:18:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary5 2024-09-22T00:18:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-22T00:18:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-22T00:18:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:18:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:18:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-22T00:18:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:18:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:18:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-22T00:18:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:18:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:18:16.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:18:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: d3b5b5e9-88cd-4138-ba00-75ea869fb1dd 2024-09-22T00:18:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:18:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2393292.8,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"} 2024-09-22T00:18:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:18:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:18:05 2024-09-22T00:18:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:18:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:18:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:18:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:18:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:18:15' 2024-09-22T00:18:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:18:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:18:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2393292.8,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"} 2024-09-22T00:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:18:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:18:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:18:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:18:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:18:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:18:05' 2024-09-22T00:18:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-22 00:18:05' 2024-09-22T00:18:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:18:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:18:05' 2024-09-22T00:18:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:18:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK542614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+replayingreplaying, {"bytes_per_second":2408584.53,"bytes_per_snapshot":72257536.0,"last_snapshot_bytes":72257536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:15image-primary6d3b5b5e9-88cd-4138-ba00-75ea869fb1ddup+replayingreplaying, {"bytes_per_second":2393292.8,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:15image-primary7f038f20d-309b-496d-b976-53bbf0460858up+replayingreplaying, {"bytes_per_second":2227677.87,"bytes_per_snapshot":66830336.0,"last_snapshot_bytes":66830336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:15image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+replayingreplaying, {"bytes_per_second":2391517.87,"bytes_per_snapshot":71745536.0,"last_snapshot_bytes":71745536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:15image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+replayingreplaying, {"bytes_per_second":2163916.8,"bytes_per_snapshot":64917504.0,"last_snapshot_bytes":64917504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:15' 2024-09-22T00:18:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:18:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:18:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:18:05' 2024-09-22T00:18:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:18:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:18:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:18:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-22T00:18:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:18:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:18:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-22T00:18:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-22T00:18:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:18:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-22T00:18:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:18:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:18:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary6 2024-09-22T00:18:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary6 2024-09-22T00:18:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-22T00:18:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-22T00:18:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:18:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:18:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-22T00:18:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:18:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:18:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-22T00:18:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:18:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:18:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:18:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f038f20d-309b-496d-b976-53bbf0460858 2024-09-22T00:18:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:18:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2227677.87,"bytes_per_snapshot":66830336.0,"last_snapshot_bytes":66830336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"} 2024-09-22T00:18:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:18:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:18:05 2024-09-22T00:18:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:18:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:18:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:18:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:18:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:18:15' 2024-09-22T00:18:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:18:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:18:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:18:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2227677.87,"bytes_per_snapshot":66830336.0,"last_snapshot_bytes":66830336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"} 2024-09-22T00:18:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:18:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:18:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:18:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:18:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:18:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:18:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:18:05' 2024-09-22T00:18:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-22 00:18:05' 2024-09-22T00:18:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:18:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:18:05' 2024-09-22T00:18:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:18:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK442614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+replayingreplaying, {"bytes_per_second":2408584.53,"bytes_per_snapshot":72257536.0,"last_snapshot_bytes":72257536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:15image-primary7f038f20d-309b-496d-b976-53bbf0460858up+replayingreplaying, {"bytes_per_second":2227677.87,"bytes_per_snapshot":66830336.0,"last_snapshot_bytes":66830336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964245,"remote_snapshot_timestamp":1726964245,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:15image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+replayingreplaying, {"bytes_per_second":2391517.87,"bytes_per_snapshot":71745536.0,"last_snapshot_bytes":71745536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:15image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+replayingreplaying, {"bytes_per_second":2163916.8,"bytes_per_snapshot":64917504.0,"last_snapshot_bytes":64917504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:15' 2024-09-22T00:18:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:18:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:18:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:18:05' 2024-09-22T00:18:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:18:19.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:18:19.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:19.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:18:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-22T00:18:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:18:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:18:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-22T00:18:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-22T00:18:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:18:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-22T00:18:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:18:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:18:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary7 2024-09-22T00:18:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary7 2024-09-22T00:18:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-22T00:18:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-22T00:18:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:18:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:18:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-22T00:18:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:18:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:18:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-22T00:18:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:18:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:18:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:18:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 85b93c29-7d43-4d0a-8feb-205586f1412b 2024-09-22T00:18:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:18:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2391517.87,"bytes_per_snapshot":71745536.0,"last_snapshot_bytes":71745536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"} 2024-09-22T00:18:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:18:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:18:05 2024-09-22T00:18:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:18:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:18:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:18:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:18:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:18:20' 2024-09-22T00:18:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:18:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:18:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:18:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2391517.87,"bytes_per_snapshot":71745536.0,"last_snapshot_bytes":71745536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"} 2024-09-22T00:18:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:18:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:18:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:18:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:18:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:18:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:18:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:18:05' 2024-09-22T00:18:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-22 00:18:05' 2024-09-22T00:18:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:18:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:18:05' 2024-09-22T00:18:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:18:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+replayingreplaying, {"bytes_per_second":2408584.53,"bytes_per_snapshot":72257536.0,"last_snapshot_bytes":72257536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:20image-primary885b93c29-7d43-4d0a-8feb-205586f1412bup+replayingreplaying, {"bytes_per_second":2391517.87,"bytes_per_snapshot":71745536.0,"last_snapshot_bytes":71745536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:20image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+replayingreplaying, {"bytes_per_second":2163916.8,"bytes_per_snapshot":64917504.0,"last_snapshot_bytes":64917504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:20' 2024-09-22T00:18:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:18:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:18:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:18:05' 2024-09-22T00:18:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:18:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:18:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:18:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-22T00:18:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:18:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:18:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-22T00:18:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-22T00:18:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:18:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-22T00:18:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:18:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:18:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary8 2024-09-22T00:18:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary8 2024-09-22T00:18:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-22T00:18:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-22T00:18:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:24.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:18:24.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:18:24.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-22T00:18:24.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:24.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:24.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:18:24.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:24.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:24.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:24.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:24.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:18:24.693 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:18:24.690+0000 7fab5fe19640 -1 reset not still connected to 0x5623f97bd860 2024-09-22T00:18:24.693 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:18:24.690+0000 7fab5fe19640 -1 reset not still connected to 0x5623f97bdba0 2024-09-22T00:18:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-22T00:18:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:18:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:18:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:18:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f3d555de-4bdb-4256-96ae-c858dd8364e9 2024-09-22T00:18:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:18:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2163916.8,"bytes_per_snapshot":64917504.0,"last_snapshot_bytes":64917504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"} 2024-09-22T00:18:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:18:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:18:05 2024-09-22T00:18:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:18:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:18:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:18:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:18:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:18:20' 2024-09-22T00:18:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:18:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:18:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:18:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2163916.8,"bytes_per_snapshot":64917504.0,"last_snapshot_bytes":64917504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"} 2024-09-22T00:18:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:18:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:18:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:18:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:18:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:18:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:18:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:18:05' 2024-09-22T00:18:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-22 00:18:05' 2024-09-22T00:18:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:18:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:18:05' 2024-09-22T00:18:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:18:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK242614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+replayingreplaying, {"bytes_per_second":2408584.53,"bytes_per_snapshot":72257536.0,"last_snapshot_bytes":72257536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:25image-primary9f3d555de-4bdb-4256-96ae-c858dd8364e9up+replayingreplaying, {"bytes_per_second":2163916.8,"bytes_per_snapshot":64917504.0,"last_snapshot_bytes":64917504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:25' 2024-09-22T00:18:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:18:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:18:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:18:05' 2024-09-22T00:18:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:18:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:18:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:18:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-22T00:18:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:18:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:18:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-22T00:18:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-22T00:18:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:18:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-22T00:18:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:18:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:18:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary9 2024-09-22T00:18:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary9 2024-09-22T00:18:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-22T00:18:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-22T00:18:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:18:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:18:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-22T00:18:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:18:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:18:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-22T00:18:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:18:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:18:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:18:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:18:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:18:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: b40c8172-0f71-438a-aaba-7aeb6f08c4a1 2024-09-22T00:18:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:18:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2408584.53,"bytes_per_snapshot":72257536.0,"last_snapshot_bytes":72257536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"} 2024-09-22T00:18:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:18:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:18:05 2024-09-22T00:18:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:18:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:18:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:18:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:18:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:18:25' 2024-09-22T00:18:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:18:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:18:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:18:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2408584.53,"bytes_per_snapshot":72257536.0,"last_snapshot_bytes":72257536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"} 2024-09-22T00:18:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:18:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:18:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:18:28.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:18:28.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:18:28.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:18:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:18:05' 2024-09-22T00:18:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-22 00:18:05' 2024-09-22T00:18:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:18:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:18:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:18:05' 2024-09-22T00:18:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:18:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:18:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:18:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK142614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10b40c8172-0f71-438a-aaba-7aeb6f08c4a1up+replayingreplaying, {"bytes_per_second":2408584.53,"bytes_per_snapshot":72257536.0,"last_snapshot_bytes":72257536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964244,"remote_snapshot_timestamp":1726964244,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:18:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:18:25' 2024-09-22T00:18:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:18:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:18:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:18:05' 2024-09-22T00:18:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:18:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:18:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:18:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:18:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:18:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-22T00:18:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:18:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:18:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-22T00:18:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-22T00:18:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:18:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-22T00:18:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:18:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:18:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:18:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary10 2024-09-22T00:18:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary10 2024-09-22T00:18:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..10} 2024-09-22T00:18:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS=() 2024-09-22T00:18:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS=() 2024-09-22T00:18:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS=() 2024-09-22T00:18:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS=() 2024-09-22T00:18:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ RET=0 2024-09-22T00:18:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-22T00:18:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary1 2024-09-22T00:18:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-22T00:18:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:18:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:18:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-22T00:18:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:18:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:18:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:18:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:18:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-22T00:18:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:18:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:18:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-22T00:18:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:18:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-22T00:18:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:18:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-22T00:18:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary1': 2024-09-22T00:18:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:18:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:18:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:18:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2287a9f423e6 2024-09-22T00:18:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2287a9f423e6 2024-09-22T00:18:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:18:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:18:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:18:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:18:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:18:30 2024 2024-09-22T00:18:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:18:30 2024 2024-09-22T00:18:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:18:30 2024 2024-09-22T00:18:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:18:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:18:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 828d093e-6c2d-483d-8b0b-98b9020322a7 2024-09-22T00:18:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:18:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:18:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:18:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-22T00:18:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd0 2024-09-22T00:18:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:18:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-22T00:18:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:18:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:18:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:18:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 93ceaa13-0e16-482a-ac92-e8e61a0f3c92 2024-09-22T00:18:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:18:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:18:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:18:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:18:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:18:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:18:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:31.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary1 2024-09-22T00:18:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-22T00:18:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:18:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary1 2024-09-22T00:18:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:18:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-22T00:18:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:18:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary1 2024-09-22T00:18:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-22T00:18:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary2 2024-09-22T00:18:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-22T00:18:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary1 2024-09-22T00:18:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-22T00:18:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:18:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:18:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-22T00:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:18:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:18:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:18:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:18:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-22T00:18:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:18:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:18:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-22T00:18:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:18:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-22T00:18:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:18:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1325 2024-09-22T00:18:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-22T00:18:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary2': 2024-09-22T00:18:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:18:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:18:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:18:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2296f1078cf9 2024-09-22T00:18:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2296f1078cf9 2024-09-22T00:18:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:18:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:18:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:18:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:18:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:18:31 2024 2024-09-22T00:18:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:18:31 2024 2024-09-22T00:18:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:18:31 2024 2024-09-22T00:18:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:18:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:18:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 93e56412-d819-4201-b562-71a8cd0af7d1 2024-09-22T00:18:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:18:32.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:18:32.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:18:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-22T00:18:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd1 2024-09-22T00:18:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:18:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-22T00:18:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:18:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:18:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:18:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: f67223a0-7f13-4a41-8504-d17d8bb17b1b 2024-09-22T00:18:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:18:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:18:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:18:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:18:32.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:18:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:18:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-22T00:18:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:18:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary2 2024-09-22T00:18:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-22T00:18:32.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:18:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary2 2024-09-22T00:18:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:18:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-22T00:18:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:18:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:18:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:18:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary2 2024-09-22T00:18:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-22T00:18:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary3 2024-09-22T00:18:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:18:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-22T00:18:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:18:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary2 2024-09-22T00:18:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:18:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-22T00:18:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:18:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-22T00:18:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:18:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:18:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:18:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:18:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-22T00:18:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:18:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:18:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-22T00:18:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:18:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-22T00:18:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:18:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-22T00:18:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:18:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1327 2024-09-22T00:18:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-22T00:18:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary3': 2024-09-22T00:18:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:18:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:18:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:18:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 22a556937662 2024-09-22T00:18:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.22a556937662 2024-09-22T00:18:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:18:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:18:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:18:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:18:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:18:32 2024 2024-09-22T00:18:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:18:32 2024 2024-09-22T00:18:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:18:32 2024 2024-09-22T00:18:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:18:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:18:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: ee43baff-84d3-43e7-a6a7-733c2d5edd84 2024-09-22T00:18:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:18:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:18:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:18:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-22T00:18:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd2 2024-09-22T00:18:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:18:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-22T00:18:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:18:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:18:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:18:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: b8add85e-bb96-4344-82f1-520997da7943 2024-09-22T00:18:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:18:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:18:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:18:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:18:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:18:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:18:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary3 2024-09-22T00:18:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-22T00:18:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:18:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary3 2024-09-22T00:18:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:18:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-22T00:18:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:18:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:18:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:18:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary3 2024-09-22T00:18:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-22T00:18:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary4 2024-09-22T00:18:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:18:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:18:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-22T00:18:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:18:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:18:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary3 2024-09-22T00:18:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-22T00:18:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-22T00:18:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:18:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:18:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:18:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:18:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-22T00:18:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:18:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:18:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-22T00:18:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:18:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-22T00:18:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:18:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-22T00:18:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:18:34.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-22T00:18:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1329 2024-09-22T00:18:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary4': 2024-09-22T00:18:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:18:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:18:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:18:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 22b4c5bf9ad3 2024-09-22T00:18:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.22b4c5bf9ad3 2024-09-22T00:18:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:18:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:18:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:18:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:18:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:18:33 2024 2024-09-22T00:18:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:18:33 2024 2024-09-22T00:18:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:18:33 2024 2024-09-22T00:18:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:18:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:18:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 6b682d11-ea96-4a51-ba17-37a405d790d5 2024-09-22T00:18:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:18:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:18:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:18:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-22T00:18:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd3 2024-09-22T00:18:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:18:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-22T00:18:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:18:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:18:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:18:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 7dbb2b6c-d1d6-47d7-96c1-140beb79e9dd 2024-09-22T00:18:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:18:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:18:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:18:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:18:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:18:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:18:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary4 2024-09-22T00:18:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-22T00:18:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:18:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary4 2024-09-22T00:18:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:18:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-22T00:18:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:18:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:18:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:18:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary4 2024-09-22T00:18:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-22T00:18:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary5 2024-09-22T00:18:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:18:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-22T00:18:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:18:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:18:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:18:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-22T00:18:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary4 2024-09-22T00:18:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-22T00:18:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:18:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:18:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:18:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:18:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-22T00:18:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:18:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:18:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-22T00:18:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:18:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-22T00:18:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:18:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-22T00:18:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:18:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1331 2024-09-22T00:18:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:18:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:18:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:18:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-22T00:18:35.526 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:18:35.522+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:18:35.526 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:18:35.522+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T00:18:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary5': 2024-09-22T00:18:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:18:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:18:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:18:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 22c2d6ba0842 2024-09-22T00:18:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.22c2d6ba0842 2024-09-22T00:18:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:18:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:18:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:18:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:18:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:18:34 2024 2024-09-22T00:18:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:18:34 2024 2024-09-22T00:18:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:18:34 2024 2024-09-22T00:18:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:18:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:18:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: e4a602d9-fa62-4577-b1fe-7fbea906e5c9 2024-09-22T00:18:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:18:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:18:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:18:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-22T00:18:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd4 2024-09-22T00:18:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:18:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-22T00:18:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:18:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:18:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:18:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 504cb3ad-806d-42eb-8fa6-2779954f69ff 2024-09-22T00:18:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:18:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:18:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:18:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:18:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:18:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:18:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary5 2024-09-22T00:18:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-22T00:18:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:18:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary5 2024-09-22T00:18:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:18:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:18:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-22T00:18:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-22T00:18:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary6 2024-09-22T00:18:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary5 2024-09-22T00:18:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-22T00:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-22T00:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:18:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary5 2024-09-22T00:18:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:18:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:18:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-22T00:18:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-22T00:18:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:18:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:18:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:18:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:18:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:18:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:18:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-22T00:18:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:18:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-22T00:18:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:18:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-22T00:18:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:18:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1334 2024-09-22T00:18:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-22T00:18:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:18:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:18:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:18:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary6': 2024-09-22T00:18:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:18:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:18:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:18:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 22d3a119dd44 2024-09-22T00:18:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.22d3a119dd44 2024-09-22T00:18:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:18:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:18:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:18:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:18:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:18:35 2024 2024-09-22T00:18:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:18:35 2024 2024-09-22T00:18:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:18:35 2024 2024-09-22T00:18:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:18:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:18:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 8c14cec8-0465-437d-9b52-6468f0ecc3a1 2024-09-22T00:18:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:18:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:18:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:18:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-22T00:18:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd5 2024-09-22T00:18:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:18:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-22T00:18:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:18:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:18:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:18:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 8f771afe-0a2f-4e13-a690-f158517b785b 2024-09-22T00:18:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:18:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:18:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:18:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:18:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:18:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:18:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary6 2024-09-22T00:18:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-22T00:18:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:18:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary6 2024-09-22T00:18:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:18:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:18:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-22T00:18:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary7 2024-09-22T00:18:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-22T00:18:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary6 2024-09-22T00:18:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-22T00:18:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:18:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:18:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-22T00:18:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:18:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:18:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:18:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:18:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary6 2024-09-22T00:18:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-22T00:18:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-22T00:18:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:18:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:18:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:18:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:18:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:18:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:18:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-22T00:18:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:18:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-22T00:18:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:18:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-22T00:18:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:18:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:18:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:18:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1333 2024-09-22T00:18:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:18:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1337 2024-09-22T00:18:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-22T00:18:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary7': 2024-09-22T00:18:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:18:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:18:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:18:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 22e4cd98ca75 2024-09-22T00:18:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.22e4cd98ca75 2024-09-22T00:18:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:18:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:18:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:18:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:18:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:18:36 2024 2024-09-22T00:18:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:18:36 2024 2024-09-22T00:18:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:18:36 2024 2024-09-22T00:18:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:18:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:18:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 35751d26-ac65-4ecd-9f96-ee35219bf69d 2024-09-22T00:18:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:18:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:18:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:18:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-22T00:18:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd6 2024-09-22T00:18:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:18:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-22T00:18:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:18:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:18:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:18:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: d54d5211-e6d4-48b0-95c2-5148d03484e3 2024-09-22T00:18:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:18:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:18:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:18:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:18:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:18:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:18:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary7 2024-09-22T00:18:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-22T00:18:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:18:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary7 2024-09-22T00:18:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:18:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-22T00:18:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:18:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-22T00:18:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary8 2024-09-22T00:18:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-22T00:18:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary7 2024-09-22T00:18:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:18:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:18:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:18:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-22T00:18:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:18:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:18:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:18:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:18:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary7 2024-09-22T00:18:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:18:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-22T00:18:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-22T00:18:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:18:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:18:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:18:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-22T00:18:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:18:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:18:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-22T00:18:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:18:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-22T00:18:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1336 2024-09-22T00:18:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:18:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:18:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:18:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:18:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-22T00:18:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1340 2024-09-22T00:18:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary8': 2024-09-22T00:18:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:18:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:18:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:18:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 22f5edfe303a 2024-09-22T00:18:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.22f5edfe303a 2024-09-22T00:18:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:18:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:18:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:18:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:18:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:18:38 2024 2024-09-22T00:18:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:18:38 2024 2024-09-22T00:18:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:18:38 2024 2024-09-22T00:18:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:18:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:18:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: fb2cac64-e4ab-4b90-81af-b904dd93e301 2024-09-22T00:18:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:18:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:18:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:18:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-22T00:18:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd7 2024-09-22T00:18:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:18:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-22T00:18:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:18:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:18:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:18:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: ba319e01-2b52-4d55-93ff-63001ff27877 2024-09-22T00:18:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:18:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:18:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:18:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:18:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:18:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:18:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:38.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary8 2024-09-22T00:18:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-22T00:18:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:18:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary8 2024-09-22T00:18:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:18:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-22T00:18:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:18:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:18:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:18:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:18:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:18:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary8 2024-09-22T00:18:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-22T00:18:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary9 2024-09-22T00:18:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-22T00:18:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary8 2024-09-22T00:18:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-22T00:18:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:18:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:18:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-22T00:18:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:18:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:18:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:18:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:18:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-22T00:18:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:18:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:18:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-22T00:18:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:18:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-22T00:18:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:18:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-22T00:18:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1339 2024-09-22T00:18:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:18:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:18:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:18:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1342 2024-09-22T00:18:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1343 2024-09-22T00:18:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:18:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-22T00:18:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary9': 2024-09-22T00:18:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:18:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:18:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:18:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 230782e72cd5 2024-09-22T00:18:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.230782e72cd5 2024-09-22T00:18:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:18:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:18:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:18:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:18:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:18:39 2024 2024-09-22T00:18:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:18:39 2024 2024-09-22T00:18:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:18:39 2024 2024-09-22T00:18:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:18:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:18:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 0eb92fe8-fe95-429d-92f9-b0269f499b8c 2024-09-22T00:18:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:18:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:18:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:18:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-22T00:18:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:18:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:18:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:18:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd8 2024-09-22T00:18:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:18:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-22T00:18:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:18:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:18:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:18:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 3f117c54-4d9b-4fb1-a144-d975363dd552 2024-09-22T00:18:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:18:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:18:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:18:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:18:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:18:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:18:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:18:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:18:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:18:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary9 2024-09-22T00:18:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-22T00:18:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:18:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary9 2024-09-22T00:18:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:18:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-22T00:18:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:18:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:18:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary9 2024-09-22T00:18:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:18:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-22T00:18:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary10 2024-09-22T00:18:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:18:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-22T00:18:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:18:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:18:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary9 2024-09-22T00:18:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:18:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-22T00:18:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:18:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-22T00:18:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:18:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:18:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:18:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:18:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:18:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-22T00:18:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:18:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-22T00:18:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:18:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:18:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:18:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-22T00:18:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:18:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:18:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-22T00:18:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1345 2024-09-22T00:18:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1348 2024-09-22T00:18:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:18:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:18:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:18:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:18:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:18:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:18:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1347 2024-09-22T00:18:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1346 2024-09-22T00:18:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:18:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:18:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:18:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:18:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-22T00:18:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary10': 2024-09-22T00:18:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:18:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:18:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:18:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 231eb549c675 2024-09-22T00:18:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.231eb549c675 2024-09-22T00:18:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:18:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:18:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:18:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:18:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:18:40 2024 2024-09-22T00:18:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:18:40 2024 2024-09-22T00:18:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:18:40 2024 2024-09-22T00:18:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:18:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:18:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: ddbf97b0-1bce-4286-9a94-25d353dc14cd 2024-09-22T00:18:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:18:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:18:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:18:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-22T00:18:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd9 2024-09-22T00:18:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:18:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-22T00:18:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:18:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:18:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:18:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 7c1148df-1f24-47e8-b743-1f6ac78cb940 2024-09-22T00:18:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:18:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:18:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:18:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:18:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:18:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:18:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:18:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary10 2024-09-22T00:18:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-22T00:18:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:18:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary10 2024-09-22T00:18:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:18:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:18:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-22T00:18:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:18:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 75591 2024-09-22T00:18:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary10 2024-09-22T00:18:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary10 2024-09-22T00:18:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-22T00:18:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:18:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:18:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:18:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:18:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:18:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-22T00:18:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1350 2024-09-22T00:18:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:18:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:18:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:18:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1354 2024-09-22T00:18:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:18:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:18:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:18:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1352 2024-09-22T00:18:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1353 2024-09-22T00:18:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:18:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:18:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:18:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:18:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:18:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:18:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1355 2024-09-22T00:18:45.117 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:18:45.114+0000 7f54f2d95640 -1 reset not still connected to 0x5617f5e6ef70 2024-09-22T00:18:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1356 2024-09-22T00:18:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:18:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:18:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:18:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:18:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:18:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:18:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1358 2024-09-22T00:18:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1357 2024-09-22T00:18:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:18:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:18:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:18:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:18:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:18:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:18:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1359 2024-09-22T00:18:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1360 2024-09-22T00:18:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:47.212 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:18:47.208+0000 7fab1f496640 -1 reset not still connected to 0x55fa64c20a90 2024-09-22T00:18:47.213 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:18:47.208+0000 7fab1f496640 -1 reset not still connected to 0x55fa67a8dee0 2024-09-22T00:18:47.213 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:18:47.208+0000 7fab1f496640 -1 reset not still connected to 0x55fa67ab4000 2024-09-22T00:18:47.213 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:18:47.208+0000 7fab1f496640 -1 reset not still connected to 0x55fa7645c270 2024-09-22T00:18:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:18:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:18:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:18:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:18:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:18:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:18:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1362 2024-09-22T00:18:48.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1361 2024-09-22T00:18:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:18:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:18:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:18:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:18:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:18:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:18:49.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1363 2024-09-22T00:18:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1364 2024-09-22T00:18:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:18:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:18:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:18:49.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:49.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:18:49.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:49.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:18:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:18:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1366 2024-09-22T00:18:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1365 2024-09-22T00:18:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:18:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:18:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:18:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:18:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:18:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:18:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:18:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:18:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:18:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:18:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:18:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:18:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1368 2024-09-22T00:18:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1367 2024-09-22T00:18:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1369 2024-09-22T00:18:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1370 2024-09-22T00:18:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:18:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:18:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:18:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:18:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:18:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:18:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1371 2024-09-22T00:18:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1372 2024-09-22T00:18:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:18:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:18:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:18:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:18:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:18:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:18:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1374 2024-09-22T00:18:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1373 2024-09-22T00:18:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:18:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:18:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:18:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:18:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:18:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:18:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1375 2024-09-22T00:18:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1376 2024-09-22T00:18:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:18:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:18:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:18:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:18:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:18:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:18:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:56.192 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:18:56.190+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3ada410 2024-09-22T00:18:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:18:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:18:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:18:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1377 2024-09-22T00:18:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1378 2024-09-22T00:18:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:18:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:18:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:18:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:18:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:18:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:18:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:18:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:18:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:18:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1380 2024-09-22T00:18:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1379 2024-09-22T00:18:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:57.285 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:18:57.282+0000 7fab5fe19640 -1 reset not still connected to 0x56241a0c2410 2024-09-22T00:18:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1382 2024-09-22T00:18:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1381 2024-09-22T00:18:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:18:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:18:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:18:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:18:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:18:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:18:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1383 2024-09-22T00:18:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1384 2024-09-22T00:18:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:18:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:18:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:18:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:18:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:18:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:18:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:18:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:18:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:18:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:19:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1386 2024-09-22T00:19:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1385 2024-09-22T00:19:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:19:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:19:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:19:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:19:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:19:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:19:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:01.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1388 2024-09-22T00:19:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1387 2024-09-22T00:19:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:19:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:19:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:19:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:19:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:19:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:19:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:19:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:19:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:19:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:19:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:19:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:19:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1390 2024-09-22T00:19:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1389 2024-09-22T00:19:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1391 2024-09-22T00:19:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1392 2024-09-22T00:19:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:19:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:19:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:19:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:19:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:19:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:19:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1393 2024-09-22T00:19:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1394 2024-09-22T00:19:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:19:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:19:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:19:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1395 2024-09-22T00:19:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1396 2024-09-22T00:19:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:19:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:19:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:19:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:19:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:19:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:19:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1397 2024-09-22T00:19:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1398 2024-09-22T00:19:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:19:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:19:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:19:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:19:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:19:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:19:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:19:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:19:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:19:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:19:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:19:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:19:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1399 2024-09-22T00:19:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1400 2024-09-22T00:19:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1401 2024-09-22T00:19:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1402 2024-09-22T00:19:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:19:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:19:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:19:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:19:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:19:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:19:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1404 2024-09-22T00:19:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1403 2024-09-22T00:19:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:19:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:19:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:19:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:19:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:19:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:19:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1406 2024-09-22T00:19:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1405 2024-09-22T00:19:10.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:19:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:19:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:19:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:19:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:19:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:19:11.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1407 2024-09-22T00:19:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:19:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:19:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:19:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:19:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:19:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:19:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1409 2024-09-22T00:19:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1408 2024-09-22T00:19:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:19:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:19:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:19:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:19:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:19:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:19:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1412 2024-09-22T00:19:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1410 2024-09-22T00:19:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:19:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:19:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:19:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:19:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1411 2024-09-22T00:19:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:19:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:19:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:13.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1413 2024-09-22T00:19:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1414 2024-09-22T00:19:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:19:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:19:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:19:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1416 2024-09-22T00:19:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:19:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:19:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:19:15.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:15.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:19:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:19:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:19:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1417 2024-09-22T00:19:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1415 2024-09-22T00:19:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:19:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:19:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:19:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:19:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:19:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:19:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:19:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:19:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:19:16.228 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:19:16.225+0000 7f54f2d95640 -1 reset not still connected to 0x5617f9b7f860 2024-09-22T00:19:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1419 2024-09-22T00:19:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:16.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:19:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:19:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:19:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1418 2024-09-22T00:19:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:19:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:19:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:19:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:17.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1420 2024-09-22T00:19:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1422 2024-09-22T00:19:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1421 2024-09-22T00:19:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:19:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:19:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:19:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1424 2024-09-22T00:19:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1423 2024-09-22T00:19:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:19:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:19:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:19:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:19:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:19:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:19:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:19:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:19:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:19:20.054 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:19:20.052+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a367be8820 2024-09-22T00:19:20.054 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:19:20.052+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a367c165b0 2024-09-22T00:19:20.054 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:19:20.052+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a369383d40 2024-09-22T00:19:20.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1425 2024-09-22T00:19:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:19:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:19:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:19:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1426 2024-09-22T00:19:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:19:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:19:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:19:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1428 2024-09-22T00:19:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1427 2024-09-22T00:19:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:19:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:19:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:19:21.245 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:19:21.240+0000 7fab1f496640 -1 reset not still connected to 0x55fa64c20a90 2024-09-22T00:19:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:19:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:19:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:19:21.246 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:19:21.244+0000 7fab1f496640 -1 reset not still connected to 0x55fa67a8dee0 2024-09-22T00:19:21.251 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:19:21.248+0000 7fab1f496640 -1 reset not still connected to 0x55fa67ab4000 2024-09-22T00:19:21.257 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:19:21.256+0000 7fab1f496640 -1 reset not still connected to 0x55fa68f93ba0 2024-09-22T00:19:21.257 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:19:21.256+0000 7fab1f496640 -1 reset not still connected to 0x55fa7645c270 2024-09-22T00:19:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:19:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:19:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:19:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:19:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:19:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:19:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1430 2024-09-22T00:19:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1429 2024-09-22T00:19:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1432 2024-09-22T00:19:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1431 2024-09-22T00:19:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:19:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:19:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:19:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:23.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:23.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:19:23.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:23.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:23.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:19:23.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:19:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1433 2024-09-22T00:19:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1434 2024-09-22T00:19:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:19:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:19:24.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:19:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:19:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:19:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:19:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1436 2024-09-22T00:19:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1435 2024-09-22T00:19:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:19:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:19:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:19:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:19:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:19:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:19:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1438 2024-09-22T00:19:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1437 2024-09-22T00:19:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:19:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:19:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:19:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:19:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:19:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:19:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:19:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:19:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:19:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:19:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:19:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:19:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1440 2024-09-22T00:19:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1439 2024-09-22T00:19:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1442 2024-09-22T00:19:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1441 2024-09-22T00:19:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:19:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:19:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:19:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:19:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:19:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:19:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1443 2024-09-22T00:19:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1444 2024-09-22T00:19:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:19:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:19:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:19:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:29.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:19:29.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:29.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:29.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:19:29.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:19:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1446 2024-09-22T00:19:30.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1445 2024-09-22T00:19:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:19:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:19:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:19:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:19:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:19:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:19:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1447 2024-09-22T00:19:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:19:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:19:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:19:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1448 2024-09-22T00:19:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:19:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:19:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:19:31.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:19:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:19:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:19:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:19:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:19:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:19:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1450 2024-09-22T00:19:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1449 2024-09-22T00:19:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:33.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1452 2024-09-22T00:19:33.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1451 2024-09-22T00:19:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:19:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:19:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:19:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:19:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:19:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:19:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1454 2024-09-22T00:19:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1453 2024-09-22T00:19:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:19:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:19:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:19:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:19:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:19:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:19:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1455 2024-09-22T00:19:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1456 2024-09-22T00:19:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:19:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:19:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:19:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:19:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:19:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:19:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1458 2024-09-22T00:19:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1457 2024-09-22T00:19:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:36.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:19:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:19:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:19:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:19:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:19:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:19:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1460 2024-09-22T00:19:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1459 2024-09-22T00:19:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:19:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:19:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:19:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:19:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:19:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:19:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1461 2024-09-22T00:19:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1462 2024-09-22T00:19:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:19:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:19:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:19:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:19:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:19:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:19:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1463 2024-09-22T00:19:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1464 2024-09-22T00:19:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:19:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:19:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:19:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:19:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:19:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:19:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1465 2024-09-22T00:19:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1466 2024-09-22T00:19:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:19:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:19:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:19:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:19:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:19:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:19:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1467 2024-09-22T00:19:41.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1468 2024-09-22T00:19:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:19:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:19:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:19:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:19:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:19:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:19:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1470 2024-09-22T00:19:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:19:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:19:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:19:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:19:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:19:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:19:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1469 2024-09-22T00:19:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1471 2024-09-22T00:19:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1472 2024-09-22T00:19:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:19:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:19:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:19:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:19:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:19:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:19:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1474 2024-09-22T00:19:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1473 2024-09-22T00:19:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:19:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:19:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:19:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:19:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:19:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:19:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1475 2024-09-22T00:19:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1476 2024-09-22T00:19:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:19:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:19:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:19:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:19:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:19:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:19:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1478 2024-09-22T00:19:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1477 2024-09-22T00:19:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:19:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:19:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:19:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:19:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:19:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:19:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:19:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:19:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:19:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:19:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:19:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:19:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1480 2024-09-22T00:19:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1479 2024-09-22T00:19:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1481 2024-09-22T00:19:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1482 2024-09-22T00:19:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:19:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:19:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:19:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:19:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:19:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:19:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1484 2024-09-22T00:19:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1483 2024-09-22T00:19:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:19:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:19:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:19:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:19:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:19:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:19:49.507 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:19:49.504+0000 7fab5fe19640 -1 reset not still connected to 0x5623fbfa9790 2024-09-22T00:19:49.508 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:19:49.504+0000 7fab5fe19640 -1 reset not still connected to 0x5623fc8c6a90 2024-09-22T00:19:49.508 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:19:49.504+0000 7fab5fe19640 -1 reset not still connected to 0x5623fde22750 2024-09-22T00:19:49.508 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:19:49.504+0000 7fab5fe19640 -1 reset not still connected to 0x56240c1e9380 2024-09-22T00:19:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1486 2024-09-22T00:19:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:19:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:19:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:19:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1485 2024-09-22T00:19:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:19:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:19:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:19:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1488 2024-09-22T00:19:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1487 2024-09-22T00:19:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:19:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:19:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:19:51.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:19:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:19:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:19:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:19:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:19:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:19:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:19:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:19:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:19:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1489 2024-09-22T00:19:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1490 2024-09-22T00:19:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1492 2024-09-22T00:19:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:19:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:19:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:19:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:19:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1491 2024-09-22T00:19:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1494 2024-09-22T00:19:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1493 2024-09-22T00:19:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:19:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:19:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:19:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:19:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:19:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:19:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1496 2024-09-22T00:19:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1495 2024-09-22T00:19:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:19:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:19:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:19:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:19:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:19:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:19:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1497 2024-09-22T00:19:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1498 2024-09-22T00:19:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:19:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:19:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:19:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:56.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:19:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:19:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:19:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:19:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:19:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:19:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:19:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:19:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:19:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1499 2024-09-22T00:19:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1500 2024-09-22T00:19:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1501 2024-09-22T00:19:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1504 2024-09-22T00:19:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:19:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:19:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:19:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:19:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:19:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:19:58.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1503 2024-09-22T00:19:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:19:59.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1502 2024-09-22T00:19:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:19:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:19:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:19:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:19:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1506 2024-09-22T00:20:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1505 2024-09-22T00:20:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:20:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:20:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:20:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:20:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:20:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:20:01.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1507 2024-09-22T00:20:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1508 2024-09-22T00:20:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:01.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:01.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:01.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:20:01.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:01.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:01.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:20:01.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:20:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:20:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:20:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:20:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1510 2024-09-22T00:20:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1509 2024-09-22T00:20:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:20:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:20:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:20:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:20:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:20:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:20:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1511 2024-09-22T00:20:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1512 2024-09-22T00:20:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:20:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:20:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:20:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:20:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:20:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:20:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:04.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1513 2024-09-22T00:20:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1514 2024-09-22T00:20:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:20:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:20:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:20:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:20:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:20:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:20:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1516 2024-09-22T00:20:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1515 2024-09-22T00:20:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:20:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:20:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:20:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:20:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:20:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:20:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1517 2024-09-22T00:20:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:20:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:20:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:20:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1518 2024-09-22T00:20:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:20:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:20:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:20:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1520 2024-09-22T00:20:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:20:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:20:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:20:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1519 2024-09-22T00:20:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:20:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:20:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:20:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1522 2024-09-22T00:20:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1521 2024-09-22T00:20:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:20:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:20:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:20:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:20:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:20:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:20:09.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1523 2024-09-22T00:20:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1524 2024-09-22T00:20:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:20:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:20:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:20:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:20:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:20:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:20:10.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1525 2024-09-22T00:20:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1526 2024-09-22T00:20:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:20:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:20:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:20:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:20:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:20:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:20:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1527 2024-09-22T00:20:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1528 2024-09-22T00:20:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:20:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:20:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:20:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:20:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:20:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:20:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1529 2024-09-22T00:20:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:20:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:20:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:20:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1530 2024-09-22T00:20:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:20:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:20:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:20:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1531 2024-09-22T00:20:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:20:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:20:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:20:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:20:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:20:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:20:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1532 2024-09-22T00:20:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1534 2024-09-22T00:20:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1533 2024-09-22T00:20:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:20:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:20:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:20:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:20:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:20:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:20:15.193 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:20:15.192+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a366bc9110 2024-09-22T00:20:15.194 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:20:15.192+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a367be8820 2024-09-22T00:20:15.194 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:20:15.192+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a367c165b0 2024-09-22T00:20:15.194 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:20:15.192+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a369383d40 2024-09-22T00:20:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1536 2024-09-22T00:20:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1535 2024-09-22T00:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:20:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:20:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:20:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:20:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:20:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1538 2024-09-22T00:20:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1537 2024-09-22T00:20:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:16.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:16.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:20:16.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:16.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:16.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:20:16.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:20:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:20:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:20:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:20:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:20:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:20:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:20:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:20:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:20:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:20:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1539 2024-09-22T00:20:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1540 2024-09-22T00:20:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1541 2024-09-22T00:20:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:20:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:20:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:20:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:20:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:20:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:20:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1543 2024-09-22T00:20:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:19.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1542 2024-09-22T00:20:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1544 2024-09-22T00:20:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:19.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:19.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:20:19.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:19.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:19.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:20:19.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:20:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:20:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:20:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:20:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1545 2024-09-22T00:20:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1546 2024-09-22T00:20:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:20:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:20:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:20:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:20.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:20.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:20:20.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:20.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:20.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:20:20.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:20:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1547 2024-09-22T00:20:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1548 2024-09-22T00:20:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:20:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:20:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:20:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:20:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:20:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:20:21.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:20:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:20:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:20:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:20:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:20:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:20:22.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1549 2024-09-22T00:20:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1550 2024-09-22T00:20:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1551 2024-09-22T00:20:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1552 2024-09-22T00:20:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:20:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:20:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:20:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:20:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:20:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:20:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1554 2024-09-22T00:20:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1553 2024-09-22T00:20:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:20:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:20:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:20:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:20:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:20:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:20:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1556 2024-09-22T00:20:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1555 2024-09-22T00:20:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:20:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:20:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:20:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:20:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:20:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:20:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1558 2024-09-22T00:20:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1557 2024-09-22T00:20:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:20:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:20:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:20:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:20:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:20:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:20:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:20:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:20:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:20:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:20:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:20:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:20:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1559 2024-09-22T00:20:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1560 2024-09-22T00:20:27.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:20:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:20:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:20:28.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1561 2024-09-22T00:20:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1562 2024-09-22T00:20:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:20:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:20:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:20:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1564 2024-09-22T00:20:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1563 2024-09-22T00:20:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:20:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:20:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:20:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:20:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:20:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:20:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1565 2024-09-22T00:20:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1566 2024-09-22T00:20:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:30.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:30.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:20:30.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:30.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:30.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:20:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:20:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:20:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:20:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:20:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1567 2024-09-22T00:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1568 2024-09-22T00:20:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:20:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:20:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:20:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:20:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:20:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:20:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1569 2024-09-22T00:20:32.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1570 2024-09-22T00:20:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:20:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:20:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:20:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:20:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:20:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:20:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1572 2024-09-22T00:20:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1571 2024-09-22T00:20:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:20:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:20:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:20:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:20:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:20:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:20:33.251 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:20:33.248+0000 7fab5fe19640 -1 reset not still connected to 0x5623fde22750 2024-09-22T00:20:33.253 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:20:33.252+0000 7fab5fe19640 -1 reset not still connected to 0x56240c1e9380 2024-09-22T00:20:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1574 2024-09-22T00:20:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1573 2024-09-22T00:20:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:20:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:20:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:20:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:20:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:20:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:20:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1576 2024-09-22T00:20:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1575 2024-09-22T00:20:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:20:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:20:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:20:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:20:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:20:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:20:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1577 2024-09-22T00:20:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:20:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:20:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:20:36.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1578 2024-09-22T00:20:36.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:20:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:20:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:20:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1580 2024-09-22T00:20:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:20:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:20:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:20:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1579 2024-09-22T00:20:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:20:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:20:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:20:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1581 2024-09-22T00:20:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1582 2024-09-22T00:20:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:20:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:20:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:20:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:20:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:20:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:20:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1583 2024-09-22T00:20:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1584 2024-09-22T00:20:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:20:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:20:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:20:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:20:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:20:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:20:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1586 2024-09-22T00:20:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:20:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:20:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:20:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:20:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:20:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:20:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1587 2024-09-22T00:20:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1588 2024-09-22T00:20:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1585 2024-09-22T00:20:41.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:41.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:20:41.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:41.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:41.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:20:41.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:20:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:20:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:20:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:20:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:20:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:20:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:20:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:20:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:20:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:20:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1590 2024-09-22T00:20:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1589 2024-09-22T00:20:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1592 2024-09-22T00:20:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1591 2024-09-22T00:20:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:20:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:20:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:20:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:44.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1594 2024-09-22T00:20:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1593 2024-09-22T00:20:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:20:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:20:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:20:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:20:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:20:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:20:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:20:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:20:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:20:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1595 2024-09-22T00:20:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:20:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:20:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:20:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:20:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:20:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:20:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1597 2024-09-22T00:20:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1598 2024-09-22T00:20:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1596 2024-09-22T00:20:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:20:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:20:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:20:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:20:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:20:46.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:20:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:20:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:20:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:20:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:20:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:20:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:20:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1599 2024-09-22T00:20:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1600 2024-09-22T00:20:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1602 2024-09-22T00:20:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1601 2024-09-22T00:20:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:20:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:20:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:20:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:20:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:20:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:20:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1604 2024-09-22T00:20:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1603 2024-09-22T00:20:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:20:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:20:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:20:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:20:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:20:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:20:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1605 2024-09-22T00:20:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1606 2024-09-22T00:20:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:20:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:20:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:20:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:20:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:20:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:20:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1607 2024-09-22T00:20:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1608 2024-09-22T00:20:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:20:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:20:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:20:51.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:51.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:20:51.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:51.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:51.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:20:51.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:20:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:20:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:20:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:20:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:20:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:20:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:20:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1610 2024-09-22T00:20:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1612 2024-09-22T00:20:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1609 2024-09-22T00:20:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1611 2024-09-22T00:20:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:20:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:20:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:20:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:20:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:20:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:20:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1614 2024-09-22T00:20:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1613 2024-09-22T00:20:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:20:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:20:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:20:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:20:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:20:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:20:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1616 2024-09-22T00:20:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1615 2024-09-22T00:20:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:20:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:20:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:20:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1618 2024-09-22T00:20:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1617 2024-09-22T00:20:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:20:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:20:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:20:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:20:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:20:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:20:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:20:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:20:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:20:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1619 2024-09-22T00:20:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:20:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:20:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:20:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1621 2024-09-22T00:20:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:20:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:20:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:20:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1620 2024-09-22T00:20:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:20:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:20:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:20:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1622 2024-09-22T00:20:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:20:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:20:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:20:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1623 2024-09-22T00:20:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1624 2024-09-22T00:20:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:20:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:20:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:20:59.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:20:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:20:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:20:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:20:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:20:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:20:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:20:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:21:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1625 2024-09-22T00:21:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1626 2024-09-22T00:21:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:21:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 75718 2024-09-22T00:21:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:21:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:21:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:21:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:21:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:21:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:21:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:21:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:21:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1627 2024-09-22T00:21:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1628 2024-09-22T00:21:01.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:21:01.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:21:01.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:21:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:21:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:21:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:21:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:21:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 75845 2024-09-22T00:21:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:21:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:21:02.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1629 2024-09-22T00:21:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:21:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:21:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:21:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:21:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:21:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:21:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:21:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 75972 2024-09-22T00:21:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:21:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1630 2024-09-22T00:21:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:21:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 76119 2024-09-22T00:21:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:21:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:21:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:21:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:21:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:21:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:21:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:21:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1631 2024-09-22T00:21:04.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:21:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:21:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:21:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:21:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:21:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:21:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:21:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:21:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 76266 2024-09-22T00:21:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1632 2024-09-22T00:21:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:21:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 76414 2024-09-22T00:21:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:21:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1633 2024-09-22T00:21:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:21:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:21:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:21:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:21:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:21:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:21:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:21:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:21:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 76562 2024-09-22T00:21:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1634 2024-09-22T00:21:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:21:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 76751 2024-09-22T00:21:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:21:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:21:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 76941 2024-09-22T00:21:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:21:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:21:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 75592 2024-09-22T00:23:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:23:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:23:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:23:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 75719 2024-09-22T00:23:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:23:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:23:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:23:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 75846 2024-09-22T00:23:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:23:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:23:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:23:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 75973 2024-09-22T00:23:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:23:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:23:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:23:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 76120 2024-09-22T00:23:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:23:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:23:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:23:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 76267 2024-09-22T00:23:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:23:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:23:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:23:36.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 76415 2024-09-22T00:23:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:23:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:23:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:23:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 76563 2024-09-22T00:23:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:23:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:23:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:23:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 76752 2024-09-22T00:23:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:23:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:23:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:23:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 76942 2024-09-22T00:23:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:23:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:23:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:23:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:23:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:23:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:23:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 1 2024-09-22T00:23:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:23:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd0 2024-09-22T00:23:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:23:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary1 2024-09-22T00:23:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary1 2024-09-22T00:23:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:23:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 2 2024-09-22T00:23:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary1 2024-09-22T00:23:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd1 2024-09-22T00:23:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:23:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary2 2024-09-22T00:23:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary2 2024-09-22T00:23:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:23:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary2 2024-09-22T00:23:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:23:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 3 2024-09-22T00:23:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd2 2024-09-22T00:23:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:23:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary3 2024-09-22T00:23:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 4 2024-09-22T00:23:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary3 2024-09-22T00:23:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:23:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:23:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary3 2024-09-22T00:23:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd3 2024-09-22T00:23:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary4 2024-09-22T00:23:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary4 2024-09-22T00:23:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:23:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary4 2024-09-22T00:23:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:23:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 5 2024-09-22T00:23:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:23:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd4 2024-09-22T00:23:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary5 2024-09-22T00:23:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary5 2024-09-22T00:23:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:23:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary5 2024-09-22T00:23:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:23:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:23:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 6 2024-09-22T00:23:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd5 2024-09-22T00:23:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:23:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:23:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 7 2024-09-22T00:23:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary6 2024-09-22T00:23:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary6 2024-09-22T00:23:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd6 2024-09-22T00:23:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary7 2024-09-22T00:23:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary7 2024-09-22T00:23:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:23:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:23:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary7 2024-09-22T00:23:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:23:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:23:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary6 2024-09-22T00:23:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:23:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:23:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:23:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:23:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 83005 2024-09-22T00:23:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 10 2024-09-22T00:23:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd9 2024-09-22T00:23:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary10 2024-09-22T00:23:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary10 2024-09-22T00:23:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:23:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary10 2024-09-22T00:23:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 9 2024-09-22T00:23:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd8 2024-09-22T00:23:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary9 2024-09-22T00:23:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary9 2024-09-22T00:23:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:23:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary9 2024-09-22T00:23:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 8 2024-09-22T00:23:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd7 2024-09-22T00:23:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary8 2024-09-22T00:23:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary8 2024-09-22T00:23:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:23:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary8 2024-09-22T00:23:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-22T00:23:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:23:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-22T00:23:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:23:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-22T00:23:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:23:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-22T00:23:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:23:43.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-22T00:23:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:23:43.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-22T00:23:43.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:23:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-22T00:23:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:23:43.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-22T00:23:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:23:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-22T00:23:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:23:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:23:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-22T00:24:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=49414032e4daae6baab47d4fb0986ae1 2024-09-22T00:24:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-22T00:24:15.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=564996227fd3d22bb797f3b075efbc69 2024-09-22T00:24:15.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-22T00:24:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-22T00:24:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:15.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:15.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:24:15.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-22T00:24:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-22T00:24:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:24:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-22T00:24:15.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=4372851c71473e0ebd68c929f2e49609 2024-09-22T00:24:15.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-22T00:24:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=df5e4c59702dba764a9ba67ddf175915 2024-09-22T00:24:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-22T00:24:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-22T00:24:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:24:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-22T00:24:16.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=87180176bedf40683960c1d02f4925f8 2024-09-22T00:24:16.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-22T00:24:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=de431a60d349c7344292a849b944de6d 2024-09-22T00:24:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-22T00:24:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=95bee0923540ca7e8f920eb11cc9a014 2024-09-22T00:24:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-22T00:24:16.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-22T00:24:16.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:16.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:24:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-22T00:24:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-22T00:24:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:24:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-22T00:24:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=d81f6522834405847f0c2b9337ee4ad9 2024-09-22T00:24:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-22T00:24:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-22T00:24:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:16.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:16.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:24:16.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-22T00:24:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-22T00:24:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:24:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-22T00:24:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-22T00:24:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:24:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-22T00:24:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=8855824a666ea1ea70921bf138782f3c 2024-09-22T00:24:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-22T00:24:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-22T00:24:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:24:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-22T00:24:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=fe1777cdcf128ebd7c55b8df31c4e6bc 2024-09-22T00:24:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-22T00:24:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-22T00:24:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:24:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-22T00:24:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:24:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:24:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-22T00:24:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:24:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-22T00:24:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:24:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:24:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:24:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-22T00:24:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:24:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:24:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:24:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:24:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:24:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:24:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-22T00:24:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:24:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-22T00:24:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:24:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-22T00:24:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:24:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-22T00:24:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:24:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:24:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-22T00:24:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:24:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-22T00:24:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:24:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:24:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-22T00:24:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:24:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-22T00:24:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:24:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:24:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-22T00:24:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:24:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:24:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: fb2cac64-e4ab-4b90-81af-b904dd93e301 2024-09-22T00:24:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:24:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:24:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:15 2024-09-22T00:24:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:24:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:18' 2024-09-22T00:24:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:24:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 93e56412-d819-4201-b562-71a8cd0af7d1 2024-09-22T00:24:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:24:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:24:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:15 2024-09-22T00:24:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:24:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:18' 2024-09-22T00:24:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:24:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:24:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:24:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:24:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-22T00:24:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:24:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:24:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-22T00:24:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:24:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:24:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-22T00:24:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:24:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:24:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-22T00:24:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:24:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:24:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-22T00:24:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:24:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:24:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-22T00:24:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:24:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:19.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:19.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:19.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:19.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:19.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:24:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-22T00:24:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-22T00:24:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:24:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:24:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:24:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:24:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:24:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8c14cec8-0465-437d-9b52-6468f0ecc3a1 2024-09-22T00:24:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:24:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:24:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:15 2024-09-22T00:24:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:24:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3600384.0,"last_snapshot_bytes":974848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964461,"remote_snapshot_timestamp":1726964461,"replay_state":"idle"} 2024-09-22T00:24:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:05' 2024-09-22T00:24:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:24:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 35751d26-ac65-4ecd-9f96-ee35219bf69d 2024-09-22T00:24:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:24:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:24:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:15 2024-09-22T00:24:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:24:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2639872.0,"last_snapshot_bytes":3706880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964463,"remote_snapshot_timestamp":1726964463,"replay_state":"idle"} 2024-09-22T00:24:19.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:05' 2024-09-22T00:24:19.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:19.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:19.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:24:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:24:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:24:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:24:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:24:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 828d093e-6c2d-483d-8b0b-98b9020322a7 2024-09-22T00:24:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:24:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:24:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:15 2024-09-22T00:24:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:24:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2564096.0,"last_snapshot_bytes":4968448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964456,"remote_snapshot_timestamp":1726964456,"replay_state":"idle"} 2024-09-22T00:24:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:05' 2024-09-22T00:24:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:24:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: ddbf97b0-1bce-4286-9a94-25d353dc14cd 2024-09-22T00:24:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:24:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:24:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:15 2024-09-22T00:24:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:24:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3489792.0,"last_snapshot_bytes":3166208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964467,"remote_snapshot_timestamp":1726964467,"replay_state":"idle"} 2024-09-22T00:24:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:05' 2024-09-22T00:24:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:24:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:24:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:24:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:24:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:24:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 0eb92fe8-fe95-429d-92f9-b0269f499b8c 2024-09-22T00:24:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:24:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:24:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:15 2024-09-22T00:24:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:24:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":450560.0,"last_snapshot_bytes":143360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964465,"remote_snapshot_timestamp":1726964465,"replay_state":"idle"} 2024-09-22T00:24:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:05' 2024-09-22T00:24:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:24:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6b682d11-ea96-4a51-ba17-37a405d790d5 2024-09-22T00:24:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:24:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:24:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:15 2024-09-22T00:24:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:24:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:19' 2024-09-22T00:24:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:24:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:24:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:24:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:24:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:24:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e4a602d9-fa62-4577-b1fe-7fbea906e5c9 2024-09-22T00:24:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:24:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:24:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:15 2024-09-22T00:24:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:24:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:19' 2024-09-22T00:24:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:24:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: ee43baff-84d3-43e7-a6a7-733c2d5edd84 2024-09-22T00:24:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:24:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:24:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:15 2024-09-22T00:24:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:24:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3143680.0,"last_snapshot_bytes":6144000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964458,"remote_snapshot_timestamp":1726964458,"replay_state":"idle"} 2024-09-22T00:24:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:05' 2024-09-22T00:24:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:24:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:24:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:24:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:24:20.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-22T00:24:20.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:20.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:20.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:24:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:24:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-22T00:24:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:24:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:24:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:24:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: fb2cac64-e4ab-4b90-81af-b904dd93e301 2024-09-22T00:24:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:24:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 93e56412-d819-4201-b562-71a8cd0af7d1 2024-09-22T00:24:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:24:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:24:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:24:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:24:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:24:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:24:20.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-22T00:24:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:24:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-22T00:24:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:24:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-22T00:24:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:24:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-22T00:24:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:24:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:24:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:24:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-22T00:24:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:24:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-22T00:24:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:24:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:24:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:24:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-22T00:24:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:24:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-22T00:24:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:24:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:24:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:24:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-22T00:24:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:24:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:24:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-22T00:24:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:24:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:24:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:24:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8c14cec8-0465-437d-9b52-6468f0ecc3a1 2024-09-22T00:24:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:24:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:24:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 35751d26-ac65-4ecd-9f96-ee35219bf69d 2024-09-22T00:24:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:21.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:21.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:21.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:21.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:24:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:24:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 828d093e-6c2d-483d-8b0b-98b9020322a7 2024-09-22T00:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:24:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 0eb92fe8-fe95-429d-92f9-b0269f499b8c 2024-09-22T00:24:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:24:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:21.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:21.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:21.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:24:21.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6b682d11-ea96-4a51-ba17-37a405d790d5 2024-09-22T00:24:21.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:21.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:24:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: ddbf97b0-1bce-4286-9a94-25d353dc14cd 2024-09-22T00:24:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:24:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:24:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: ee43baff-84d3-43e7-a6a7-733c2d5edd84 2024-09-22T00:24:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:24:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:24:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:24:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e4a602d9-fa62-4577-b1fe-7fbea906e5c9 2024-09-22T00:24:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:24:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:24:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:21.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:21.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:21.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:24:21.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:21.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:24:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-22T00:24:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:24:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:24:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:24:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-22T00:24:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:24:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:24:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:24:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-22T00:24:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:24:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:24:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:24:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-22T00:24:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:24:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:24:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:24:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:24:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:24:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:24:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:24:21.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:24:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:24:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:24:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-22T00:24:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:24:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-22T00:24:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:24:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-22T00:24:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:24:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-22T00:24:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:24:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-22T00:24:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:24:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-22T00:24:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-22T00:24:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:24:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-22T00:24:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:24:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 93e56412-d819-4201-b562-71a8cd0af7d1 2024-09-22T00:24:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:24:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:24:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: fb2cac64-e4ab-4b90-81af-b904dd93e301 2024-09-22T00:24:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:24:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:22.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:22.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:22.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:22.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:22.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:22.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:22.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:22.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:22.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:22.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:22.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:22.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:22.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:22.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:22.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:22.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:22.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:22.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:22.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:22.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:24:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:24:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:24:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:24:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:24:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:24:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-22T00:24:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:24:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:24:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-22T00:24:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-22T00:24:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:24:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:24:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-22T00:24:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-22T00:24:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:24:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-22T00:24:22.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-22T00:24:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:24:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-22T00:24:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:24:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8c14cec8-0465-437d-9b52-6468f0ecc3a1 2024-09-22T00:24:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:24:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:22.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:22.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:24:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-22T00:24:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:24:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-22T00:24:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:24:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 35751d26-ac65-4ecd-9f96-ee35219bf69d 2024-09-22T00:24:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:24:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:24:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-22T00:24:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:24:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-22T00:24:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:24:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 0eb92fe8-fe95-429d-92f9-b0269f499b8c 2024-09-22T00:24:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:24:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:22.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:22.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:22.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:24:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:24:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:24:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-22T00:24:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:24:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-22T00:24:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:24:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-22T00:24:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:24:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-22T00:24:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-22T00:24:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:24:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-22T00:24:22.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:22.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:22.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:24:22.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:24:22.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-22T00:24:22.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:22.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-22T00:24:22.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:22.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-22T00:24:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-22T00:24:22.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:24:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:24:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-22T00:24:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:24:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:24:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-22T00:24:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:24:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: ddbf97b0-1bce-4286-9a94-25d353dc14cd 2024-09-22T00:24:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:24:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:23.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:23.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:23.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:24:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:24:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: ee43baff-84d3-43e7-a6a7-733c2d5edd84 2024-09-22T00:24:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:24:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:23.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:23.037 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:24:23.033+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a37a6e25b0 2024-09-22T00:24:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:24:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:24:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6b682d11-ea96-4a51-ba17-37a405d790d5 2024-09-22T00:24:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:24:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:24:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:24:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e4a602d9-fa62-4577-b1fe-7fbea906e5c9 2024-09-22T00:24:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:24:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:24:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:24:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-22T00:24:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:24:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:24:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-22T00:24:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:24:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:24:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 828d093e-6c2d-483d-8b0b-98b9020322a7 2024-09-22T00:24:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20 2024-09-22T00:24:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:24:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:24:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:24:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:20' 2024-09-22T00:24:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:24:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:24:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:24:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:24:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:23.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:24:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:24:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-22T00:24:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-22 00:24:20' 2024-09-22T00:24:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:24:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:24:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:20' 2024-09-22T00:24:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:24:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:24:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:24:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:24:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-22T00:24:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:24:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:24:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-22T00:24:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:24:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:24:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-22T00:24:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:24:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:24:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-22T00:24:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:24:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:24:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-22T00:24:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd0 2024-09-22T00:24:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:24:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-22T00:24:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:24:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:24:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:24:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:24:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-22T00:24:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:24:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:24:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-22T00:24:23.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:23.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:23.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:23.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:23.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:23.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-22T00:24:23.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:23.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:23.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:24:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:24:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-22T00:24:23.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:24:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:24:20' 2024-09-22T00:24:23.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:24:23.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:24:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:20' 2024-09-22T00:24:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:24:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:24:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:24:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:24:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:24:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:24:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-22T00:24:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:24:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:24:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:24:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-22T00:24:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd1 2024-09-22T00:24:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:24:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-22T00:24:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:24:24.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:24:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:24:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:24:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:24:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:24:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:24:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:24:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:24:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:24:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:24:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:24:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-22T00:24:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-22T00:24:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:24:24.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:24:24.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:24:24.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:24:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:24:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:24:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-22T00:24:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-22T00:24:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:24:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:24:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:24:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:24:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-22T00:24:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-22T00:24:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:24:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:24:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-22T00:24:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-22T00:24:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd2 2024-09-22T00:24:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-22T00:24:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd6 2024-09-22T00:24:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd5 2024-09-22T00:24:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-22T00:24:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-22T00:24:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd7 2024-09-22T00:24:24.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-22T00:24:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:24:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:24:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:24:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd3 2024-09-22T00:24:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-22T00:24:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd4 2024-09-22T00:24:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:24:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:24:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:24:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-22T00:24:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd8 2024-09-22T00:24:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-22T00:24:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:24:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd9 2024-09-22T00:24:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-22T00:24:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:24:25.648 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:24:25.647+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:24:25.649 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:24:25.647+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T00:24:25.653 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:24:25.651+0000 7f54f2d95640 -1 reset not still connected to 0x5618075d2b60 2024-09-22T00:24:25.653 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:24:25.651+0000 7f54f2d95640 -1 reset not still connected to 0x5618075d3110 2024-09-22T00:24:25.653 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:24:25.651+0000 7f54f2d95640 -1 reset not still connected to 0x5618075d31e0 2024-09-22T00:24:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=49414032e4daae6baab47d4fb0986ae1 2024-09-22T00:24:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-22T00:24:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 49414032e4daae6baab47d4fb0986ae1 != \4\9\4\1\4\0\3\2\e\4\d\a\a\e\6\b\a\a\b\4\7\d\4\f\b\0\9\8\6\a\e\1 ]] 2024-09-22T00:24:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=d81f6522834405847f0c2b9337ee4ad9 2024-09-22T00:24:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-22T00:24:57.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ d81f6522834405847f0c2b9337ee4ad9 != \d\8\1\f\6\5\2\2\8\3\4\4\0\5\8\4\7\f\0\c\2\b\9\3\3\7\e\e\4\a\d\9 ]] 2024-09-22T00:24:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=564996227fd3d22bb797f3b075efbc69 2024-09-22T00:24:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-22T00:24:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=8855824a666ea1ea70921bf138782f3c 2024-09-22T00:24:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-22T00:24:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 564996227fd3d22bb797f3b075efbc69 != \5\6\4\9\9\6\2\2\7\f\d\3\d\2\2\b\b\7\9\7\f\3\b\0\7\5\e\f\b\c\6\9 ]] 2024-09-22T00:24:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=4372851c71473e0ebd68c929f2e49609 2024-09-22T00:24:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-22T00:24:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 8855824a666ea1ea70921bf138782f3c != \8\8\5\5\8\2\4\a\6\6\6\e\a\1\e\a\7\0\9\2\1\b\f\1\3\8\7\8\2\f\3\c ]] 2024-09-22T00:24:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 4372851c71473e0ebd68c929f2e49609 != \4\3\7\2\8\5\1\c\7\1\4\7\3\e\0\e\b\d\6\8\c\9\2\9\f\2\e\4\9\6\0\9 ]] 2024-09-22T00:24:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=df5e4c59702dba764a9ba67ddf175915 2024-09-22T00:24:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-22T00:24:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=fe1777cdcf128ebd7c55b8df31c4e6bc 2024-09-22T00:24:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-22T00:24:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=de431a60d349c7344292a849b944de6d 2024-09-22T00:24:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-22T00:24:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=87180176bedf40683960c1d02f4925f8 2024-09-22T00:24:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-22T00:24:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ df5e4c59702dba764a9ba67ddf175915 != \d\f\5\e\4\c\5\9\7\0\2\d\b\a\7\6\4\a\9\b\a\6\7\d\d\f\1\7\5\9\1\5 ]] 2024-09-22T00:24:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:24:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 83006 2024-09-22T00:24:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:24:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 83007 2024-09-22T00:24:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ fe1777cdcf128ebd7c55b8df31c4e6bc != \f\e\1\7\7\7\c\d\c\f\1\2\8\e\b\d\7\c\5\5\b\8\d\f\3\1\c\4\e\6\b\c ]] 2024-09-22T00:24:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ de431a60d349c7344292a849b944de6d != \d\e\4\3\1\a\6\0\d\3\4\9\c\7\3\4\4\2\9\2\a\8\4\9\b\9\4\4\d\e\6\d ]] 2024-09-22T00:24:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 87180176bedf40683960c1d02f4925f8 != \8\7\1\8\0\1\7\6\b\e\d\f\4\0\6\8\3\9\6\0\c\1\d\0\2\f\4\9\2\5\f\8 ]] 2024-09-22T00:24:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=95bee0923540ca7e8f920eb11cc9a014 2024-09-22T00:24:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-22T00:24:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 95bee0923540ca7e8f920eb11cc9a014 != \9\5\b\e\e\0\9\2\3\5\4\0\c\a\7\e\8\f\9\2\0\e\b\1\1\c\c\9\a\0\1\4 ]] 2024-09-22T00:24:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:24:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 83010 2024-09-22T00:24:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:24:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 83012 2024-09-22T00:24:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:24:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 83014 2024-09-22T00:24:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:24:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 83016 2024-09-22T00:24:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:24:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 83017 2024-09-22T00:24:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:24:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 83019 2024-09-22T00:24:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:24:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 83021 2024-09-22T00:24:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:24:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:24:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-22T00:24:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-22T00:24:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:24:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:24:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-22T00:24:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:24:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:24:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:24:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:24:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-22T00:24:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:24:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:24:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:24:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:24:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:24:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:24:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 828d093e-6c2d-483d-8b0b-98b9020322a7 2024-09-22T00:24:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:24:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":30984192.0,"bytes_per_snapshot":61968384.0,"last_snapshot_bytes":61968384,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"} 2024-09-22T00:24:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:24:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:35 2024-09-22T00:24:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:24:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:24:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:24:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:24:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:55' 2024-09-22T00:24:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:24:59.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:24:59.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:24:59.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":30984192.0,"bytes_per_snapshot":61968384.0,"last_snapshot_bytes":61968384,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"} 2024-09-22T00:24:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:24:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:24:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:24:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:24:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:24:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:24:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:24:59.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:24:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:35' 2024-09-22T00:24:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-22 00:24:35' 2024-09-22T00:24:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:24:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:24:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:24:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:24:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:35' 2024-09-22T00:24:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:24:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:24:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:25:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1828d093e-6c2d-483d-8b0b-98b9020322a7up+replayingreplaying, {"bytes_per_second":30984192.0,"bytes_per_snapshot":61968384.0,"last_snapshot_bytes":61968384,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:24:55image-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+replayingreplaying, {"bytes_per_second":31619072.0,"bytes_per_snapshot":63238144.0,"last_snapshot_bytes":63238144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:24:55image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+replayingreplaying, {"bytes_per_second":32362496.0,"bytes_per_snapshot":64724992.0,"last_snapshot_bytes":64724992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964663,"remote_snapshot_timestamp":1726964663,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:24:55image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+replayingreplaying, {"bytes_per_second":31373312.0,"bytes_per_snapshot":62746624.0,"last_snapshot_bytes":62746624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:24:55image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72257536.0,"last_snapshot_bytes":72257536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:24:55image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63242240.0,"last_snapshot_bytes":63242240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:24:55image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63492096.0,"last_snapshot_bytes":63492096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:24:55image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+replayingreplaying, {"bytes_per_second":31782912.0,"bytes_per_snapshot":63565824.0,"last_snapshot_bytes":63565824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:24:55image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61186048.0,"last_snapshot_bytes":61186048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964663,"remote_snapshot_timestamp":1726964663,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:24:55image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64745472.0,"last_snapshot_bytes":64745472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:24:55' 2024-09-22T00:25:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:25:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:25:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:35' 2024-09-22T00:25:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:25:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:25:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:25:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-22T00:25:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:25:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:25:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-22T00:25:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-22T00:25:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:25:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-22T00:25:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:25:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:25:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary1 2024-09-22T00:25:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary1 2024-09-22T00:25:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-22T00:25:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-22T00:25:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:25:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:25:02.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-22T00:25:02.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:02.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:02.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:25:02.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:02.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:25:02.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:25:02.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:02.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:25:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-22T00:25:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:25:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:25:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:25:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:25:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:25:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:25:03.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:25:03.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 93e56412-d819-4201-b562-71a8cd0af7d1 2024-09-22T00:25:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:25:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":32362496.0,"bytes_per_snapshot":64724992.0,"last_snapshot_bytes":64724992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964663,"remote_snapshot_timestamp":1726964663,"replay_state":"idle"} 2024-09-22T00:25:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:25:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:24:35 2024-09-22T00:25:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:25:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:25:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:25:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:25:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:25:00' 2024-09-22T00:25:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:25:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:25:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:25:03.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":32362496.0,"bytes_per_snapshot":64724992.0,"last_snapshot_bytes":64724992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964663,"remote_snapshot_timestamp":1726964663,"replay_state":"idle"} 2024-09-22T00:25:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:25:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:25:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:25:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:25:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:25:03.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:25:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:35' 2024-09-22T00:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-22 00:24:35' 2024-09-22T00:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:24:35' 2024-09-22T00:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:25:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK942614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+replayingreplaying, {"bytes_per_second":31619072.0,"bytes_per_snapshot":63238144.0,"last_snapshot_bytes":63238144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:00image-primary293e56412-d819-4201-b562-71a8cd0af7d1up+replayingreplaying, {"bytes_per_second":32362496.0,"bytes_per_snapshot":64724992.0,"last_snapshot_bytes":64724992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964663,"remote_snapshot_timestamp":1726964663,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:00image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+replayingreplaying, {"bytes_per_second":31373312.0,"bytes_per_snapshot":62746624.0,"last_snapshot_bytes":62746624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:00image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72257536.0,"last_snapshot_bytes":72257536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:00image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63242240.0,"last_snapshot_bytes":63242240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:00image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63492096.0,"last_snapshot_bytes":63492096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:00image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+replayingreplaying, {"bytes_per_second":31782912.0,"bytes_per_snapshot":63565824.0,"last_snapshot_bytes":63565824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:00image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61186048.0,"last_snapshot_bytes":61186048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964663,"remote_snapshot_timestamp":1726964663,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:00image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64745472.0,"last_snapshot_bytes":64745472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:24:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:00' 2024-09-22T00:25:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:25:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:25:04.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:24:35' 2024-09-22T00:25:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:25:04.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:25:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:25:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-22T00:25:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:25:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:25:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-22T00:25:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-22T00:25:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:25:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-22T00:25:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:25:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:25:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary2 2024-09-22T00:25:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary2 2024-09-22T00:25:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-22T00:25:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-22T00:25:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:25:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:25:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-22T00:25:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:25:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:25:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:25:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:25:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-22T00:25:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:25:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:25:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:25:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:25:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:25:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:25:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:25:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: ee43baff-84d3-43e7-a6a7-733c2d5edd84 2024-09-22T00:25:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:25:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2091554.13,"bytes_per_snapshot":62746624.0,"last_snapshot_bytes":62746624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"} 2024-09-22T00:25:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:25:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:25:05 2024-09-22T00:25:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:25:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:25:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:25:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:25:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:25:05' 2024-09-22T00:25:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:25:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:25:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:25:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2091554.13,"bytes_per_snapshot":62746624.0,"last_snapshot_bytes":62746624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"} 2024-09-22T00:25:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:25:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:25:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:25:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:25:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:25:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:25:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:25:05' 2024-09-22T00:25:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-22 00:25:05' 2024-09-22T00:25:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:25:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:25:05' 2024-09-22T00:25:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:25:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:25:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:25:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK842614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+replayingreplaying, {"bytes_per_second":2107938.13,"bytes_per_snapshot":63238144.0,"last_snapshot_bytes":63238144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:05image-primary3ee43baff-84d3-43e7-a6a7-733c2d5edd84up+replayingreplaying, {"bytes_per_second":2091554.13,"bytes_per_snapshot":62746624.0,"last_snapshot_bytes":62746624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:05image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+replayingreplaying, {"bytes_per_second":2408584.53,"bytes_per_snapshot":72257536.0,"last_snapshot_bytes":72257536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:05image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+replayingreplaying, {"bytes_per_second":2108074.67,"bytes_per_snapshot":63242240.0,"last_snapshot_bytes":63242240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:05image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+replayingreplaying, {"bytes_per_second":2116403.2,"bytes_per_snapshot":63492096.0,"last_snapshot_bytes":63492096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:05image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+replayingreplaying, {"bytes_per_second":2118860.8,"bytes_per_snapshot":63565824.0,"last_snapshot_bytes":63565824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:05image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+replayingreplaying, {"bytes_per_second":2039534.93,"bytes_per_snapshot":61186048.0,"last_snapshot_bytes":61186048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964663,"remote_snapshot_timestamp":1726964663,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:05image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+replayingreplaying, {"bytes_per_second":2158182.4,"bytes_per_snapshot":64745472.0,"last_snapshot_bytes":64745472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:05' 2024-09-22T00:25:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:25:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:25:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:25:05' 2024-09-22T00:25:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:25:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:25:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:25:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-22T00:25:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:25:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:25:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-22T00:25:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-22T00:25:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:25:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-22T00:25:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:25:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:25:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary3 2024-09-22T00:25:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary3 2024-09-22T00:25:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-22T00:25:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-22T00:25:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:25:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:25:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-22T00:25:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:25:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:25:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:25:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:25:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-22T00:25:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:25:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:25:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:25:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:25:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:25:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:25:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:25:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6b682d11-ea96-4a51-ba17-37a405d790d5 2024-09-22T00:25:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:25:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2408584.53,"bytes_per_snapshot":72257536.0,"last_snapshot_bytes":72257536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"} 2024-09-22T00:25:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:25:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:25:05 2024-09-22T00:25:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:25:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:25:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:25:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:25:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:25:05' 2024-09-22T00:25:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:25:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:25:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:25:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2408584.53,"bytes_per_snapshot":72257536.0,"last_snapshot_bytes":72257536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"} 2024-09-22T00:25:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:25:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:25:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:25:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:25:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:25:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:25:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:25:05' 2024-09-22T00:25:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-22 00:25:05' 2024-09-22T00:25:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:25:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:25:05' 2024-09-22T00:25:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:25:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:25:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:25:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK742614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+replayingreplaying, {"bytes_per_second":2107938.13,"bytes_per_snapshot":63238144.0,"last_snapshot_bytes":63238144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:05image-primary46b682d11-ea96-4a51-ba17-37a405d790d5up+replayingreplaying, {"bytes_per_second":2408584.53,"bytes_per_snapshot":72257536.0,"last_snapshot_bytes":72257536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:05image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+replayingreplaying, {"bytes_per_second":2108074.67,"bytes_per_snapshot":63242240.0,"last_snapshot_bytes":63242240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:05image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+replayingreplaying, {"bytes_per_second":2116403.2,"bytes_per_snapshot":63492096.0,"last_snapshot_bytes":63492096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:05image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+replayingreplaying, {"bytes_per_second":2118860.8,"bytes_per_snapshot":63565824.0,"last_snapshot_bytes":63565824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:05image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+replayingreplaying, {"bytes_per_second":2039534.93,"bytes_per_snapshot":61186048.0,"last_snapshot_bytes":61186048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964663,"remote_snapshot_timestamp":1726964663,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:05image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+replayingreplaying, {"bytes_per_second":2158182.4,"bytes_per_snapshot":64745472.0,"last_snapshot_bytes":64745472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:05' 2024-09-22T00:25:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:25:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:25:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:25:05' 2024-09-22T00:25:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:25:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:25:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:25:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-22T00:25:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:25:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:25:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-22T00:25:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-22T00:25:10.034 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:25:10.027+0000 7fab5fe19640 -1 reset not still connected to 0x562402327d40 2024-09-22T00:25:10.037 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:25:10.035+0000 7fab5fe19640 -1 reset not still connected to 0x562421f04ea0 2024-09-22T00:25:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:25:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-22T00:25:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:25:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:25:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary4 2024-09-22T00:25:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary4 2024-09-22T00:25:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-22T00:25:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-22T00:25:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:25:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:25:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-22T00:25:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:25:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:25:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:25:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:25:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-22T00:25:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:25:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:25:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:25:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:25:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:25:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:25:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:25:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e4a602d9-fa62-4577-b1fe-7fbea906e5c9 2024-09-22T00:25:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:25:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2108074.67,"bytes_per_snapshot":63242240.0,"last_snapshot_bytes":63242240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"} 2024-09-22T00:25:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:25:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:25:05 2024-09-22T00:25:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:25:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:25:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:25:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:25:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:25:10' 2024-09-22T00:25:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:25:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:25:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:25:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2108074.67,"bytes_per_snapshot":63242240.0,"last_snapshot_bytes":63242240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"} 2024-09-22T00:25:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:25:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:25:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:25:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:25:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:25:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:25:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:25:05' 2024-09-22T00:25:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-22 00:25:05' 2024-09-22T00:25:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:25:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:25:05' 2024-09-22T00:25:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:25:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:25:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:25:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK642614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+replayingreplaying, {"bytes_per_second":2107938.13,"bytes_per_snapshot":63238144.0,"last_snapshot_bytes":63238144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:10image-primary5e4a602d9-fa62-4577-b1fe-7fbea906e5c9up+replayingreplaying, {"bytes_per_second":2108074.67,"bytes_per_snapshot":63242240.0,"last_snapshot_bytes":63242240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:10image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+replayingreplaying, {"bytes_per_second":2116403.2,"bytes_per_snapshot":63492096.0,"last_snapshot_bytes":63492096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:10image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+replayingreplaying, {"bytes_per_second":2118860.8,"bytes_per_snapshot":63565824.0,"last_snapshot_bytes":63565824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:10image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+replayingreplaying, {"bytes_per_second":2039534.93,"bytes_per_snapshot":61186048.0,"last_snapshot_bytes":61186048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964663,"remote_snapshot_timestamp":1726964663,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:10image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+replayingreplaying, {"bytes_per_second":2158182.4,"bytes_per_snapshot":64745472.0,"last_snapshot_bytes":64745472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:10' 2024-09-22T00:25:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:25:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:25:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:25:05' 2024-09-22T00:25:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:25:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:25:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:25:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-22T00:25:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:25:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:25:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-22T00:25:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-22T00:25:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:25:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-22T00:25:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:25:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:25:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary5 2024-09-22T00:25:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary5 2024-09-22T00:25:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-22T00:25:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-22T00:25:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:25:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:25:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-22T00:25:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:25:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:25:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:25:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:25:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-22T00:25:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:25:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:25:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:25:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:25:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:25:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:25:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:25:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8c14cec8-0465-437d-9b52-6468f0ecc3a1 2024-09-22T00:25:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:25:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2116403.2,"bytes_per_snapshot":63492096.0,"last_snapshot_bytes":63492096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"} 2024-09-22T00:25:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:25:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:25:05 2024-09-22T00:25:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:25:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:25:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:25:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:25:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:25:15' 2024-09-22T00:25:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:25:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:25:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:25:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2116403.2,"bytes_per_snapshot":63492096.0,"last_snapshot_bytes":63492096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"} 2024-09-22T00:25:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:25:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:25:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:25:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:25:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:25:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:25:16.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:25:05' 2024-09-22T00:25:16.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-22 00:25:05' 2024-09-22T00:25:16.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:16.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:16.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:25:16.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:16.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:25:05' 2024-09-22T00:25:16.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:25:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:25:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:25:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK542614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+replayingreplaying, {"bytes_per_second":2107938.13,"bytes_per_snapshot":63238144.0,"last_snapshot_bytes":63238144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:15image-primary68c14cec8-0465-437d-9b52-6468f0ecc3a1up+replayingreplaying, {"bytes_per_second":2116403.2,"bytes_per_snapshot":63492096.0,"last_snapshot_bytes":63492096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:15image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+replayingreplaying, {"bytes_per_second":2118860.8,"bytes_per_snapshot":63565824.0,"last_snapshot_bytes":63565824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:15image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+replayingreplaying, {"bytes_per_second":2039534.93,"bytes_per_snapshot":61186048.0,"last_snapshot_bytes":61186048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964663,"remote_snapshot_timestamp":1726964663,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:15image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+replayingreplaying, {"bytes_per_second":2158182.4,"bytes_per_snapshot":64745472.0,"last_snapshot_bytes":64745472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:15' 2024-09-22T00:25:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:25:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:25:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:25:05' 2024-09-22T00:25:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:25:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:25:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:25:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-22T00:25:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:25:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:25:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-22T00:25:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-22T00:25:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:25:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-22T00:25:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:25:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:25:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary6 2024-09-22T00:25:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary6 2024-09-22T00:25:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-22T00:25:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-22T00:25:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:25:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:25:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-22T00:25:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:25:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:25:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:25:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:25:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-22T00:25:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:25:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:25:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:25:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:25:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:25:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:25:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:25:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 35751d26-ac65-4ecd-9f96-ee35219bf69d 2024-09-22T00:25:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:25:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2118860.8,"bytes_per_snapshot":63565824.0,"last_snapshot_bytes":63565824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"} 2024-09-22T00:25:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:25:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:25:05 2024-09-22T00:25:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:25:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:25:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:25:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:25:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:25:15' 2024-09-22T00:25:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:25:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:25:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:25:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2118860.8,"bytes_per_snapshot":63565824.0,"last_snapshot_bytes":63565824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"} 2024-09-22T00:25:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:25:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:25:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:25:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:25:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:25:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:25:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:25:05' 2024-09-22T00:25:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-22 00:25:05' 2024-09-22T00:25:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:25:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:25:05' 2024-09-22T00:25:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:25:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:25:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:25:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK442614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+replayingreplaying, {"bytes_per_second":2107938.13,"bytes_per_snapshot":63238144.0,"last_snapshot_bytes":63238144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:15image-primary735751d26-ac65-4ecd-9f96-ee35219bf69dup+replayingreplaying, {"bytes_per_second":2118860.8,"bytes_per_snapshot":63565824.0,"last_snapshot_bytes":63565824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:15image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+replayingreplaying, {"bytes_per_second":2039534.93,"bytes_per_snapshot":61186048.0,"last_snapshot_bytes":61186048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964663,"remote_snapshot_timestamp":1726964663,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:15image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+replayingreplaying, {"bytes_per_second":2158182.4,"bytes_per_snapshot":64745472.0,"last_snapshot_bytes":64745472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:15' 2024-09-22T00:25:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:25:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:25:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:25:05' 2024-09-22T00:25:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:25:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:25:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:19.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:25:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-22T00:25:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:25:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:25:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-22T00:25:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-22T00:25:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:25:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-22T00:25:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:25:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:25:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary7 2024-09-22T00:25:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary7 2024-09-22T00:25:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-22T00:25:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-22T00:25:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:25:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:25:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-22T00:25:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:25:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:25:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:25:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:25:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-22T00:25:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:25:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:25:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:25:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:25:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:25:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:25:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:25:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: fb2cac64-e4ab-4b90-81af-b904dd93e301 2024-09-22T00:25:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:25:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2039534.93,"bytes_per_snapshot":61186048.0,"last_snapshot_bytes":61186048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964663,"remote_snapshot_timestamp":1726964663,"replay_state":"idle"} 2024-09-22T00:25:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:25:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:25:05 2024-09-22T00:25:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:25:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:25:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:25:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:25:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:25:20' 2024-09-22T00:25:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:25:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:25:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:25:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2039534.93,"bytes_per_snapshot":61186048.0,"last_snapshot_bytes":61186048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964663,"remote_snapshot_timestamp":1726964663,"replay_state":"idle"} 2024-09-22T00:25:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:25:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:25:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:25:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:25:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:25:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:25:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:25:05' 2024-09-22T00:25:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-22 00:25:05' 2024-09-22T00:25:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:25:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:25:05' 2024-09-22T00:25:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:25:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:25:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:25:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+replayingreplaying, {"bytes_per_second":2107938.13,"bytes_per_snapshot":63238144.0,"last_snapshot_bytes":63238144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:20image-primary8fb2cac64-e4ab-4b90-81af-b904dd93e301up+replayingreplaying, {"bytes_per_second":2039534.93,"bytes_per_snapshot":61186048.0,"last_snapshot_bytes":61186048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964663,"remote_snapshot_timestamp":1726964663,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:20image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+replayingreplaying, {"bytes_per_second":2158182.4,"bytes_per_snapshot":64745472.0,"last_snapshot_bytes":64745472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:20' 2024-09-22T00:25:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:25:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:25:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:25:05' 2024-09-22T00:25:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:25:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:25:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:25:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-22T00:25:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:25:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:25:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-22T00:25:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-22T00:25:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:25:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-22T00:25:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:25:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:25:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary8 2024-09-22T00:25:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary8 2024-09-22T00:25:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-22T00:25:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-22T00:25:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:25:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:25:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-22T00:25:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:25:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:25:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:25:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:25:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-22T00:25:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:25:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:25:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:25:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:25:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:25:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:25:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:25:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 0eb92fe8-fe95-429d-92f9-b0269f499b8c 2024-09-22T00:25:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:25:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2158182.4,"bytes_per_snapshot":64745472.0,"last_snapshot_bytes":64745472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"} 2024-09-22T00:25:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:25:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:25:05 2024-09-22T00:25:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:25:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:25:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:25:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:25:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:25:25' 2024-09-22T00:25:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:25:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:25:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:25:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2158182.4,"bytes_per_snapshot":64745472.0,"last_snapshot_bytes":64745472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"} 2024-09-22T00:25:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:25:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:25:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:25:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:25:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:25:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:25:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:25:05' 2024-09-22T00:25:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-22 00:25:05' 2024-09-22T00:25:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:25:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:25:05' 2024-09-22T00:25:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:25:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:25:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:25:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK242614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+replayingreplaying, {"bytes_per_second":2107938.13,"bytes_per_snapshot":63238144.0,"last_snapshot_bytes":63238144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:25image-primary90eb92fe8-fe95-429d-92f9-b0269f499b8cup+replayingreplaying, {"bytes_per_second":2158182.4,"bytes_per_snapshot":64745472.0,"last_snapshot_bytes":64745472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:25' 2024-09-22T00:25:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:25:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:25:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:25:05' 2024-09-22T00:25:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:25:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:25:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:25:25.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-22T00:25:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:25:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:25:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-22T00:25:25.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-22T00:25:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:25:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-22T00:25:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:25:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:25:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary9 2024-09-22T00:25:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary9 2024-09-22T00:25:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-22T00:25:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-22T00:25:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:25:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:25:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-22T00:25:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:25:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:25:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:25:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:25:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-22T00:25:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:25:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:25:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:25:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:25:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:25:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:25:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:25:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: ddbf97b0-1bce-4286-9a94-25d353dc14cd 2024-09-22T00:25:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:25:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2107938.13,"bytes_per_snapshot":63238144.0,"last_snapshot_bytes":63238144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"} 2024-09-22T00:25:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:25:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:25:05 2024-09-22T00:25:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:25:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:25:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:25:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:25:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:25:25' 2024-09-22T00:25:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:25:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:25:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:25:28.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2107938.13,"bytes_per_snapshot":63238144.0,"last_snapshot_bytes":63238144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"} 2024-09-22T00:25:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:25:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:25:28.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:25:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:25:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:25:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:25:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:25:05' 2024-09-22T00:25:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-22 00:25:05' 2024-09-22T00:25:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:25:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:25:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:25:05' 2024-09-22T00:25:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:25:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:25:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:25:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK142614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10ddbf97b0-1bce-4286-9a94-25d353dc14cdup+replayingreplaying, {"bytes_per_second":2107938.13,"bytes_per_snapshot":63238144.0,"last_snapshot_bytes":63238144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964664,"remote_snapshot_timestamp":1726964664,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:25:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:25:25' 2024-09-22T00:25:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:25:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:25:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:25:05' 2024-09-22T00:25:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:25:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:25:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:25:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:25:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:25:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-22T00:25:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:25:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:25:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-22T00:25:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-22T00:25:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:25:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-22T00:25:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:25:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:25:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:25:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary10 2024-09-22T00:25:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary10 2024-09-22T00:25:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..10} 2024-09-22T00:25:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS=() 2024-09-22T00:25:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS=() 2024-09-22T00:25:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS=() 2024-09-22T00:25:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS=() 2024-09-22T00:25:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ RET=0 2024-09-22T00:25:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-22T00:25:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary1 2024-09-22T00:25:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-22T00:25:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:25:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:25:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-22T00:25:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:25:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:25:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:25:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:25:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-22T00:25:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:25:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:25:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-22T00:25:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:25:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-22T00:25:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:25:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-22T00:25:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary1': 2024-09-22T00:25:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:25:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:25:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:25:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 26f98c060668 2024-09-22T00:25:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.26f98c060668 2024-09-22T00:25:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:25:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:25:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:25:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:25:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:25:30 2024 2024-09-22T00:25:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:25:30 2024 2024-09-22T00:25:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:25:30 2024 2024-09-22T00:25:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:25:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:25:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9e2e9109-8e5b-413c-8411-9ba5ae31be5f 2024-09-22T00:25:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:25:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:25:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:25:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-22T00:25:31.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd0 2024-09-22T00:25:31.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:25:31.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-22T00:25:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:25:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:25:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:25:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: a035977d-8b12-404d-8b66-c5b85ed8fb99 2024-09-22T00:25:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:25:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:25:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:25:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:25:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:25:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:25:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary1 2024-09-22T00:25:31.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-22T00:25:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:25:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary1 2024-09-22T00:25:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:25:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-22T00:25:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:25:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:25:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-22T00:25:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary1 2024-09-22T00:25:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary2 2024-09-22T00:25:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:25:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-22T00:25:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:25:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:25:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:25:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:25:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-22T00:25:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary1 2024-09-22T00:25:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-22T00:25:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:25:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:25:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:25:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:25:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-22T00:25:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:25:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:25:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-22T00:25:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:25:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-22T00:25:32.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1656 2024-09-22T00:25:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:25:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-22T00:25:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary2': 2024-09-22T00:25:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:25:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:25:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:25:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2707fc121c99 2024-09-22T00:25:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2707fc121c99 2024-09-22T00:25:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:25:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:25:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:25:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:25:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:25:31 2024 2024-09-22T00:25:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:25:31 2024 2024-09-22T00:25:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:25:31 2024 2024-09-22T00:25:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:25:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:25:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: b7c8b91b-78db-4a95-a282-7a3cffdfd580 2024-09-22T00:25:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:25:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:25:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:25:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-22T00:25:32.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd1 2024-09-22T00:25:32.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:25:32.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-22T00:25:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:25:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:25:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:25:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 0ecb7670-e8e9-4ed2-bc8f-08807f812866 2024-09-22T00:25:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:25:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:25:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:25:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:25:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:25:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:25:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary2 2024-09-22T00:25:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-22T00:25:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:25:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-22T00:25:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:25:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary2 2024-09-22T00:25:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:25:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-22T00:25:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:25:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:25:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:25:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-22T00:25:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary3 2024-09-22T00:25:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:25:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-22T00:25:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary2 2024-09-22T00:25:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:25:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:25:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:25:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-22T00:25:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:25:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:25:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:25:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:25:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary2 2024-09-22T00:25:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-22T00:25:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-22T00:25:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:25:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:25:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-22T00:25:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:25:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-22T00:25:32.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:25:32.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-22T00:25:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1658 2024-09-22T00:25:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:25:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-22T00:25:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary3': 2024-09-22T00:25:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:25:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:25:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:25:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2717e37ff25f 2024-09-22T00:25:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2717e37ff25f 2024-09-22T00:25:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:25:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:25:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:25:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:25:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:25:32 2024 2024-09-22T00:25:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:25:32 2024 2024-09-22T00:25:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:25:32 2024 2024-09-22T00:25:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:25:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:25:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 75ab6c51-4a42-443e-a205-28783bbf691c 2024-09-22T00:25:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:25:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:25:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:25:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-22T00:25:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd2 2024-09-22T00:25:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:25:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-22T00:25:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:25:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:25:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:25:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 35c18b73-775d-45f5-a486-d9c43f3cb596 2024-09-22T00:25:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:25:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:25:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:25:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:25:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:25:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:25:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary3 2024-09-22T00:25:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-22T00:25:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:25:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary3 2024-09-22T00:25:33.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:25:33.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-22T00:25:33.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:33.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:33.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:25:33.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:25:33.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:33.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:33.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary3 2024-09-22T00:25:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:25:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-22T00:25:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary4 2024-09-22T00:25:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:25:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-22T00:25:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:25:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary3 2024-09-22T00:25:33.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-22T00:25:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:25:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:25:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-22T00:25:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:25:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:25:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:25:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:25:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-22T00:25:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:25:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:25:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-22T00:25:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:25:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-22T00:25:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:25:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-22T00:25:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:25:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1660 2024-09-22T00:25:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-22T00:25:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary4': 2024-09-22T00:25:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:25:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:25:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:25:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2725c31b4716 2024-09-22T00:25:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2725c31b4716 2024-09-22T00:25:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:25:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:25:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:25:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:25:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:25:33 2024 2024-09-22T00:25:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:25:33 2024 2024-09-22T00:25:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:25:33 2024 2024-09-22T00:25:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:25:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:25:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: b5dc0f6c-1ed1-4c7a-a2ae-86146375d79e 2024-09-22T00:25:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:25:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:25:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:25:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-22T00:25:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd3 2024-09-22T00:25:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:25:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-22T00:25:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:25:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:25:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:25:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: a32d5c0a-a8c9-4802-8b6e-f609216639b3 2024-09-22T00:25:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:25:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:25:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:25:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:25:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:25:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:25:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary4 2024-09-22T00:25:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-22T00:25:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:25:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary4 2024-09-22T00:25:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:25:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-22T00:25:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:25:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:25:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:25:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-22T00:25:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:25:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary5 2024-09-22T00:25:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:25:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-22T00:25:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:25:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:25:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-22T00:25:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary4 2024-09-22T00:25:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:25:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:25:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:25:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:25:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-22T00:25:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary4 2024-09-22T00:25:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-22T00:25:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:25:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:25:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-22T00:25:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:25:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-22T00:25:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:25:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-22T00:25:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1662 2024-09-22T00:25:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:25:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:25:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:25:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:25:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-22T00:25:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary5': 2024-09-22T00:25:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:25:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:25:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:25:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2733b9830d38 2024-09-22T00:25:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2733b9830d38 2024-09-22T00:25:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:25:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:25:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:25:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:25:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:25:34 2024 2024-09-22T00:25:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:25:34 2024 2024-09-22T00:25:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:25:34 2024 2024-09-22T00:25:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:25:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:25:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 8da3684b-bb34-4654-9135-6980fd409d92 2024-09-22T00:25:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:25:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:25:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:25:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-22T00:25:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd4 2024-09-22T00:25:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:25:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-22T00:25:35.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:25:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:25:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:25:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: e2e7da4d-096a-40e0-aa48-23d5e7e825bd 2024-09-22T00:25:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:25:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:25:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:25:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:25:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:25:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:25:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary5 2024-09-22T00:25:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-22T00:25:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:25:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary5 2024-09-22T00:25:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:25:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-22T00:25:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:25:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:25:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:25:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:25:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:25:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary5 2024-09-22T00:25:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-22T00:25:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary6 2024-09-22T00:25:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-22T00:25:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary5 2024-09-22T00:25:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-22T00:25:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:25:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:25:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-22T00:25:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:25:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:25:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:25:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:25:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-22T00:25:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:25:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:25:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-22T00:25:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:25:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-22T00:25:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:25:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-22T00:25:36.257 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:25:36.253+0000 7f84953c5640 -1 reset not still connected to 0x5636f85f04e0 2024-09-22T00:25:36.257 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:25:36.253+0000 7f84953c5640 -1 reset not still connected to 0x5636f85f0a90 2024-09-22T00:25:36.258 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:25:36.257+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b332b0 2024-09-22T00:25:36.258 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:25:36.257+0000 7f84953c5640 -1 reset not still connected to 0x5636f8ec80d0 2024-09-22T00:25:36.270 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:25:36.269+0000 7f84953c5640 -1 reset not still connected to 0x5636f9c289c0 2024-09-22T00:25:36.271 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:25:36.269+0000 7f84953c5640 -1 reset not still connected to 0x5636fa732c30 2024-09-22T00:25:36.273 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:25:36.269+0000 7f84953c5640 -1 reset not still connected to 0x5636fd2c6750 2024-09-22T00:25:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:25:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1665 2024-09-22T00:25:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-22T00:25:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:25:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:25:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:25:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary6': 2024-09-22T00:25:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:25:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:25:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:25:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 27446f62d13a 2024-09-22T00:25:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.27446f62d13a 2024-09-22T00:25:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:25:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:25:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:25:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:25:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:25:35 2024 2024-09-22T00:25:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:25:35 2024 2024-09-22T00:25:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:25:35 2024 2024-09-22T00:25:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:25:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:25:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 3623ca88-98f0-4728-9ca6-b784db8b4075 2024-09-22T00:25:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:25:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:25:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:25:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-22T00:25:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd5 2024-09-22T00:25:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:25:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-22T00:25:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:25:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:25:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:25:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 2ea7f99c-c8fa-4576-bc09-539ea833e998 2024-09-22T00:25:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:25:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:25:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:25:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:25:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:25:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:25:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary6 2024-09-22T00:25:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-22T00:25:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:25:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary6 2024-09-22T00:25:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:25:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-22T00:25:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:25:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:25:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:25:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:25:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:25:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-22T00:25:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary7 2024-09-22T00:25:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-22T00:25:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:25:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:25:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-22T00:25:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary6 2024-09-22T00:25:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:25:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:25:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:25:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:25:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-22T00:25:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary6 2024-09-22T00:25:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-22T00:25:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:25:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:25:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-22T00:25:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:25:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-22T00:25:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:25:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-22T00:25:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1664 2024-09-22T00:25:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:25:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:25:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:25:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1668 2024-09-22T00:25:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:25:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-22T00:25:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary7': 2024-09-22T00:25:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:25:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:25:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:25:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 27542e1744e5 2024-09-22T00:25:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.27542e1744e5 2024-09-22T00:25:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:25:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:25:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:25:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:25:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:25:36 2024 2024-09-22T00:25:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:25:36 2024 2024-09-22T00:25:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:25:36 2024 2024-09-22T00:25:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:25:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:25:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 375b2e3e-524e-40f4-a41f-19efac72a51a 2024-09-22T00:25:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:25:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:25:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:25:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-22T00:25:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd6 2024-09-22T00:25:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:25:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-22T00:25:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:25:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:25:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:25:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 319a5b20-6181-4a07-bbc5-fe71ddba0c84 2024-09-22T00:25:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:25:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:25:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:25:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:25:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:25:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:25:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary7 2024-09-22T00:25:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-22T00:25:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:25:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary7 2024-09-22T00:25:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:25:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-22T00:25:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:25:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:25:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:25:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:25:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:25:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary7 2024-09-22T00:25:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-22T00:25:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary8 2024-09-22T00:25:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-22T00:25:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary7 2024-09-22T00:25:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-22T00:25:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:25:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:25:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-22T00:25:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:25:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:25:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:25:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:25:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-22T00:25:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:25:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:25:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-22T00:25:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:25:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-22T00:25:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:25:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-22T00:25:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1667 2024-09-22T00:25:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:25:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:25:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:25:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:25:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1671 2024-09-22T00:25:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-22T00:25:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary8': 2024-09-22T00:25:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:25:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:25:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:25:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 276682b8e9b 2024-09-22T00:25:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.276682b8e9b 2024-09-22T00:25:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:25:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:25:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:25:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:25:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:25:37 2024 2024-09-22T00:25:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:25:37 2024 2024-09-22T00:25:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:25:37 2024 2024-09-22T00:25:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:25:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:25:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 6d02ae97-ffa3-463b-bb5f-e1ee89d8d122 2024-09-22T00:25:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:25:38.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:25:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:25:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-22T00:25:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd7 2024-09-22T00:25:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:25:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-22T00:25:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:25:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:25:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:25:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 6482be29-33ba-4967-8ed6-da2a6804e337 2024-09-22T00:25:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:25:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:25:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:25:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:25:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:25:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:25:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary8 2024-09-22T00:25:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-22T00:25:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:25:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary8 2024-09-22T00:25:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:25:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:25:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-22T00:25:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary9 2024-09-22T00:25:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-22T00:25:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary8 2024-09-22T00:25:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:25:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:25:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-22T00:25:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-22T00:25:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:25:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:25:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:25:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:25:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-22T00:25:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary8 2024-09-22T00:25:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-22T00:25:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:25:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:25:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:25:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:25:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:25:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:25:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-22T00:25:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:25:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-22T00:25:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:25:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-22T00:25:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1670 2024-09-22T00:25:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1674 2024-09-22T00:25:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:25:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:25:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:25:39.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:25:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-22T00:25:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary9': 2024-09-22T00:25:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:25:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:25:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:25:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2776df63aa90 2024-09-22T00:25:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2776df63aa90 2024-09-22T00:25:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:25:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:25:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:25:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:25:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:25:38 2024 2024-09-22T00:25:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:25:38 2024 2024-09-22T00:25:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:25:38 2024 2024-09-22T00:25:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:25:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:25:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: e11fe15e-b1f5-4761-b5de-fa0922f20c54 2024-09-22T00:25:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:25:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:25:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:25:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-22T00:25:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd8 2024-09-22T00:25:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:25:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-22T00:25:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:25:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:25:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:25:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 0e53cf9f-1035-45ef-befa-79293171b47e 2024-09-22T00:25:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:25:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:25:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:25:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:25:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:25:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:25:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary9 2024-09-22T00:25:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-22T00:25:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:25:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary9 2024-09-22T00:25:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:25:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-22T00:25:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:25:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:25:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:25:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:25:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-22T00:25:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary10 2024-09-22T00:25:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:25:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-22T00:25:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:25:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:25:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary9 2024-09-22T00:25:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:25:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-22T00:25:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:25:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:25:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:25:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:25:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:25:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-22T00:25:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary9 2024-09-22T00:25:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-22T00:25:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:25:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:25:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:25:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-22T00:25:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:25:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:25:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-22T00:25:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:25:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-22T00:25:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1673 2024-09-22T00:25:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:25:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:25:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:25:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:25:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:25:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:25:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:25:40.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-22T00:25:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary10': 2024-09-22T00:25:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:25:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:25:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:25:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2786296a6127 2024-09-22T00:25:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2786296a6127 2024-09-22T00:25:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:25:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:25:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:25:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:25:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:25:39 2024 2024-09-22T00:25:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:25:39 2024 2024-09-22T00:25:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:25:39 2024 2024-09-22T00:25:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:25:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:25:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 39691262-dc09-459e-a706-083041ddeb1b 2024-09-22T00:25:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:25:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:25:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:25:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-22T00:25:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd9 2024-09-22T00:25:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:25:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-22T00:25:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:25:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:25:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:25:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 32104600-e64b-4202-be15-78e9be08861a 2024-09-22T00:25:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:25:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:25:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:25:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:25:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:25:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:25:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:25:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary10 2024-09-22T00:25:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-22T00:25:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:25:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary10 2024-09-22T00:25:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:25:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-22T00:25:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:25:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:25:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary10 2024-09-22T00:25:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:25:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 86684 2024-09-22T00:25:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:25:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:25:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:25:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary10 2024-09-22T00:25:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-22T00:25:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:25:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-22T00:25:41.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1676 2024-09-22T00:25:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1678 2024-09-22T00:25:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:25:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:25:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:25:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1681 2024-09-22T00:25:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:25:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:25:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:25:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1680 2024-09-22T00:25:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1679 2024-09-22T00:25:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:25:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:25:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:25:42.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:25:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:25:42.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:25:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1682 2024-09-22T00:25:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1683 2024-09-22T00:25:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:25:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:25:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:25:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1685 2024-09-22T00:25:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1684 2024-09-22T00:25:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:25:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:25:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:25:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:25:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:25:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:25:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:25:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:25:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:25:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1686 2024-09-22T00:25:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:25:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:25:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:25:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:25:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:25:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:25:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1687 2024-09-22T00:25:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1688 2024-09-22T00:25:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1689 2024-09-22T00:25:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:25:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:25:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:25:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:25:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:25:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:25:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:25:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:25:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:25:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:25:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:25:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:25:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1690 2024-09-22T00:25:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1691 2024-09-22T00:25:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1692 2024-09-22T00:25:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1693 2024-09-22T00:25:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:25:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:25:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:25:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1695 2024-09-22T00:25:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1694 2024-09-22T00:25:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:49.318 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:25:49.314+0000 7fab5fe19640 -1 reset not still connected to 0x56242475c750 2024-09-22T00:25:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:25:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:25:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:25:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:25:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:25:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:25:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:25:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:25:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:25:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1696 2024-09-22T00:25:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:25:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:25:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:25:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:25:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:25:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:25:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:50.454 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:25:50.446+0000 7f54f2d95640 -1 reset not still connected to 0x5617f324f450 2024-09-22T00:25:50.454 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:25:50.446+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:25:50.454 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:25:50.446+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T00:25:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1699 2024-09-22T00:25:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1698 2024-09-22T00:25:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1697 2024-09-22T00:25:51.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:51.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:51.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:25:51.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:51.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:51.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:25:51.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:25:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:51.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:25:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:25:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:25:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:25:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:25:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:25:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:25:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:25:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:25:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1700 2024-09-22T00:25:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1701 2024-09-22T00:25:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:52.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:53.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1702 2024-09-22T00:25:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1703 2024-09-22T00:25:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:25:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:25:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:25:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:53.908 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:25:53.902+0000 7fab5fe19640 -1 reset not still connected to 0x5623f976e4e0 2024-09-22T00:25:53.908 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:25:53.902+0000 7fab5fe19640 -1 reset not still connected to 0x5623f97bd860 2024-09-22T00:25:53.913 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:25:53.910+0000 7fab5fe19640 -1 reset not still connected to 0x5623f97bdba0 2024-09-22T00:25:53.913 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:25:53.910+0000 7fab5fe19640 -1 reset not still connected to 0x562407fa6d00 2024-09-22T00:25:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1705 2024-09-22T00:25:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1704 2024-09-22T00:25:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:25:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:25:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:25:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:25:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:25:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:25:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:25:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:25:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:25:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1706 2024-09-22T00:25:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:25:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:25:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:25:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:25:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:25:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:25:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1707 2024-09-22T00:25:56.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1709 2024-09-22T00:25:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1708 2024-09-22T00:25:56.407 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:25:56.398+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7d991ab60 2024-09-22T00:25:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:25:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:25:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:25:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:25:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:25:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:25:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:25:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:25:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:25:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:25:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:25:57.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:25:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1710 2024-09-22T00:25:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1711 2024-09-22T00:25:57.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1712 2024-09-22T00:25:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1713 2024-09-22T00:25:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:25:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:25:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:25:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1715 2024-09-22T00:25:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1714 2024-09-22T00:25:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:59.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:25:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:25:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:25:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:25:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:25:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:25:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:25:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:25:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:25:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:25:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:25:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:25:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:26:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1716 2024-09-22T00:26:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:26:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:26:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:26:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:26:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:26:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:26:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1717 2024-09-22T00:26:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1718 2024-09-22T00:26:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1719 2024-09-22T00:26:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:26:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:26:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:26:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:26:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:26:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:26:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:01.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:26:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:26:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:26:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:26:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:26:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:26:02.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1721 2024-09-22T00:26:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1722 2024-09-22T00:26:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1720 2024-09-22T00:26:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1723 2024-09-22T00:26:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:26:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:26:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:26:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1724 2024-09-22T00:26:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1725 2024-09-22T00:26:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:04.306 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:26:04.302+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3393110 2024-09-22T00:26:04.306 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:26:04.302+0000 7f54f2d95640 -1 reset not still connected to 0x5617f33931e0 2024-09-22T00:26:04.315 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:26:04.306+0000 7f54f2d95640 -1 reset not still connected to 0x561805c96820 2024-09-22T00:26:04.315 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:26:04.306+0000 7f54f2d95640 -1 reset not still connected to 0x5618092c2820 2024-09-22T00:26:04.315 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:26:04.306+0000 7f54f2d95640 -1 reset not still connected to 0x56181219af70 2024-09-22T00:26:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:26:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:26:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:26:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:26:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:26:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:26:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:26:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:26:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:26:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1726 2024-09-22T00:26:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:26:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:26:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:26:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1729 2024-09-22T00:26:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1728 2024-09-22T00:26:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1727 2024-09-22T00:26:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:26:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:26:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:26:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:26:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:26:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:26:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:26:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:26:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:26:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1730 2024-09-22T00:26:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:26:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:26:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:26:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1732 2024-09-22T00:26:07.301 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:26:07.294+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7e07cc680 2024-09-22T00:26:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:26:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:26:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:26:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1731 2024-09-22T00:26:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1733 2024-09-22T00:26:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:26:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:26:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:26:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:08.336 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:26:08.330+0000 7fab5fe19640 -1 reset not still connected to 0x56240a858dd0 2024-09-22T00:26:08.340 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:26:08.338+0000 7fab5fe19640 -1 reset not still connected to 0x56241af7ed00 2024-09-22T00:26:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1734 2024-09-22T00:26:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1735 2024-09-22T00:26:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:26:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:26:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:26:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:26:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:26:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:26:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:26:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:26:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:26:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1736 2024-09-22T00:26:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:26:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:26:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:26:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:26:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:26:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:26:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1738 2024-09-22T00:26:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1737 2024-09-22T00:26:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:26:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:26:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:26:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:26:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:26:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:26:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1739 2024-09-22T00:26:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1740 2024-09-22T00:26:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1741 2024-09-22T00:26:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:26:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:26:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:26:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:26:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:26:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:26:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1742 2024-09-22T00:26:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1743 2024-09-22T00:26:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:26:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:26:13.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:26:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1745 2024-09-22T00:26:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1744 2024-09-22T00:26:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:26:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:26:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:26:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:26:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:26:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:26:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1746 2024-09-22T00:26:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:26:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:26:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:26:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:26:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:26:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:26:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1749 2024-09-22T00:26:16.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1747 2024-09-22T00:26:16.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1748 2024-09-22T00:26:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:26:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:26:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:26:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:26:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:26:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:26:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:17.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1751 2024-09-22T00:26:17.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:17.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:26:17.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:26:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:26:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:26:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:26:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:26:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1750 2024-09-22T00:26:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1752 2024-09-22T00:26:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1753 2024-09-22T00:26:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:26:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:26:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:26:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1754 2024-09-22T00:26:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1755 2024-09-22T00:26:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:26:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:26:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:26:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:26:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:26:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:26:19.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:26:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:26:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:26:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1756 2024-09-22T00:26:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:26:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:26:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:26:20.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:20.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:26:20.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:20.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:20.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:26:20.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:26:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1757 2024-09-22T00:26:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1758 2024-09-22T00:26:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1759 2024-09-22T00:26:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:26:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:26:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:26:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:26:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:26:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:26:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:26:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:26:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:26:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:26:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:26:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:26:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1761 2024-09-22T00:26:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1760 2024-09-22T00:26:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1762 2024-09-22T00:26:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:26:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:26:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:26:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1763 2024-09-22T00:26:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1765 2024-09-22T00:26:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1764 2024-09-22T00:26:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:24.374 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:26:24.372+0000 7f84953c5640 -1 reset not still connected to 0x5636fbd20750 2024-09-22T00:26:24.374 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:26:24.372+0000 7f84953c5640 -1 reset not still connected to 0x5636fd2c6750 2024-09-22T00:26:24.374 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:26:24.372+0000 7f84953c5640 -1 reset not still connected to 0x5637038e4820 2024-09-22T00:26:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:26:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:26:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:26:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:26:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:26:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:26:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:26:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:26:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:26:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1766 2024-09-22T00:26:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:26:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:26:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:26:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:26:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1769 2024-09-22T00:26:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1767 2024-09-22T00:26:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1768 2024-09-22T00:26:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:26:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:26:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:26:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:26:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:26:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:26:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:26:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:26:27.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:26:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:26:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:26:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:26:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1771 2024-09-22T00:26:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1770 2024-09-22T00:26:27.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1772 2024-09-22T00:26:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:26:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:26:28.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:26:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1775 2024-09-22T00:26:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1774 2024-09-22T00:26:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1773 2024-09-22T00:26:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:26:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:26:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:26:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:26:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:26:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:26:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:26:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:26:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:26:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1776 2024-09-22T00:26:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:26:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:26:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:26:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:26:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:26:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:26:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1778 2024-09-22T00:26:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1777 2024-09-22T00:26:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1779 2024-09-22T00:26:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:26:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:26:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:26:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:26:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:26:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:26:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1781 2024-09-22T00:26:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1780 2024-09-22T00:26:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:26:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:26:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:26:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:26:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:26:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:26:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1783 2024-09-22T00:26:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1782 2024-09-22T00:26:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:26:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:26:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:26:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1784 2024-09-22T00:26:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1785 2024-09-22T00:26:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:26:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:26:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:26:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:26:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:26:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:26:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:26:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:26:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1786 2024-09-22T00:26:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:26:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:26:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:26:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:26:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:26:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:26:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1787 2024-09-22T00:26:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1788 2024-09-22T00:26:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1789 2024-09-22T00:26:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:26:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:36.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:26:36.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:26:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:26:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1790 2024-09-22T00:26:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:26:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:26:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:26:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:26:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:26:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:26:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1791 2024-09-22T00:26:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1793 2024-09-22T00:26:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:26:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:26:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:26:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1792 2024-09-22T00:26:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1794 2024-09-22T00:26:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1795 2024-09-22T00:26:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:39.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:26:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:26:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:26:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:26:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:26:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:26:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:26:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:26:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:26:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1796 2024-09-22T00:26:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:26:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:26:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:26:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:26:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:26:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:26:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1798 2024-09-22T00:26:41.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1799 2024-09-22T00:26:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1797 2024-09-22T00:26:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:26:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:26:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:26:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:26:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:26:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:26:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:26:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:26:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:26:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:26:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:26:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:26:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1800 2024-09-22T00:26:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1801 2024-09-22T00:26:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1803 2024-09-22T00:26:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1802 2024-09-22T00:26:43.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:26:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:26:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:26:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1805 2024-09-22T00:26:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1804 2024-09-22T00:26:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:26:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:26:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:26:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:26:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:26:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:26:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:26:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:26:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:26:44.407 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:26:44.400+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a3630960d0 2024-09-22T00:26:44.408 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:26:44.400+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a364135ba0 2024-09-22T00:26:44.408 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:26:44.400+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a3661cba00 2024-09-22T00:26:44.409 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:26:44.400+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a36720c8f0 2024-09-22T00:26:44.409 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:26:44.400+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a36bae45b0 2024-09-22T00:26:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1806 2024-09-22T00:26:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:26:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:26:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:26:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:26:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:26:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:26:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1808 2024-09-22T00:26:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1809 2024-09-22T00:26:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1807 2024-09-22T00:26:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:26:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:26:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:26:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:26:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:26:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:26:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:26:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:26:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:26:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:26:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:26:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:26:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1811 2024-09-22T00:26:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1810 2024-09-22T00:26:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:26:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:26:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:26:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1813 2024-09-22T00:26:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1812 2024-09-22T00:26:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1815 2024-09-22T00:26:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1814 2024-09-22T00:26:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:26:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:26:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:26:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:26:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:26:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:26:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:26:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:26:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:26:49.631 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:26:49.625+0000 7fab5fe19640 -1 reset not still connected to 0x5623fd2fbe10 2024-09-22T00:26:49.632 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:26:49.629+0000 7fab5fe19640 -1 reset not still connected to 0x5623fd831110 2024-09-22T00:26:49.632 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:26:49.629+0000 7fab5fe19640 -1 reset not still connected to 0x56240a858dd0 2024-09-22T00:26:49.632 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:26:49.629+0000 7fab5fe19640 -1 reset not still connected to 0x56241af7ed00 2024-09-22T00:26:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:26:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:26:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:26:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:26:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:26:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1819 2024-09-22T00:26:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1816 2024-09-22T00:26:51.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1818 2024-09-22T00:26:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1817 2024-09-22T00:26:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:26:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:26:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:26:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:26:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:26:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:26:51.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:26:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:26:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:26:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:26:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:26:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:26:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1820 2024-09-22T00:26:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1821 2024-09-22T00:26:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1822 2024-09-22T00:26:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1825 2024-09-22T00:26:54.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1824 2024-09-22T00:26:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1823 2024-09-22T00:26:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:54.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:26:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:26:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:26:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:26:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:26:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:26:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:26:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:26:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:26:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:26:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:26:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:26:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:26:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:26:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:26:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:26:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:26:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:26:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1828 2024-09-22T00:26:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1826 2024-09-22T00:26:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1827 2024-09-22T00:26:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:26:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:26:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:26:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:26:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:26:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:26:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:26:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:26:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:26:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:26:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:26:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:26:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1829 2024-09-22T00:26:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1831 2024-09-22T00:26:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1832 2024-09-22T00:26:58.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1830 2024-09-22T00:26:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:26:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:26:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:26:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1834 2024-09-22T00:26:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1835 2024-09-22T00:26:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1833 2024-09-22T00:26:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:26:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:26:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:26:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:26:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:26:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:26:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:26:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:26:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:26:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:27:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1836 2024-09-22T00:27:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:27:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:27:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:27:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:27:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:27:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:27:01.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1838 2024-09-22T00:27:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1837 2024-09-22T00:27:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:01.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1840 2024-09-22T00:27:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:27:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:27:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:27:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:27:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:27:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:27:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:27:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:27:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:27:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1839 2024-09-22T00:27:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:27:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:27:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:27:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1841 2024-09-22T00:27:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:27:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:27:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:27:02.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1843 2024-09-22T00:27:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1842 2024-09-22T00:27:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:27:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:27:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:27:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:03.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1845 2024-09-22T00:27:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1844 2024-09-22T00:27:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:27:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:27:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:27:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:27:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:27:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:27:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:27:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:27:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:27:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1846 2024-09-22T00:27:05.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:05.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:27:05.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:05.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:05.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:27:05.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:27:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:27:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:27:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:27:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1849 2024-09-22T00:27:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1847 2024-09-22T00:27:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1848 2024-09-22T00:27:06.320 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:06.316+0000 7f84953c5640 -1 reset not still connected to 0x5636f85f0a90 2024-09-22T00:27:06.320 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:06.316+0000 7f84953c5640 -1 reset not still connected to 0x5636f85f0d00 2024-09-22T00:27:06.320 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:06.316+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b32ea0 2024-09-22T00:27:06.320 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:06.316+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b332b0 2024-09-22T00:27:06.320 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:06.316+0000 7f84953c5640 -1 reset not still connected to 0x5636f8ec80d0 2024-09-22T00:27:06.320 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:06.316+0000 7f84953c5640 -1 reset not still connected to 0x5636f9c289c0 2024-09-22T00:27:06.320 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:06.316+0000 7f84953c5640 -1 reset not still connected to 0x5636fd2c6750 2024-09-22T00:27:06.320 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:06.316+0000 7f84953c5640 -1 reset not still connected to 0x5637038e4820 2024-09-22T00:27:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:27:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:27:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:27:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:27:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:27:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:27:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:06.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1851 2024-09-22T00:27:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1850 2024-09-22T00:27:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:27:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:27:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:27:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:27:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:27:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:27:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1853 2024-09-22T00:27:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1852 2024-09-22T00:27:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:27:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:27:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:27:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1854 2024-09-22T00:27:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1855 2024-09-22T00:27:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:27:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:27:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:27:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:27:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:27:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:27:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:27:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:27:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:27:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1856 2024-09-22T00:27:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:27:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:27:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:27:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:27:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:27:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:27:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1857 2024-09-22T00:27:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1858 2024-09-22T00:27:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1859 2024-09-22T00:27:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:27:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:27:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:27:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:27:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:27:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:27:11.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:27:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:27:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:27:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:27:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:27:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:27:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1861 2024-09-22T00:27:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1860 2024-09-22T00:27:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:13.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1863 2024-09-22T00:27:13.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1862 2024-09-22T00:27:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:27:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:27:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:27:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1865 2024-09-22T00:27:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1864 2024-09-22T00:27:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:27:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:27:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:27:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:27:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:27:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:27:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:27:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:27:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:27:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1866 2024-09-22T00:27:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:27:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:27:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:27:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:27:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:27:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:27:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1869 2024-09-22T00:27:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1867 2024-09-22T00:27:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1868 2024-09-22T00:27:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:27:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:27:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:27:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:27:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:27:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:27:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:27:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:27:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:27:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:27:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:27:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:27:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1871 2024-09-22T00:27:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1870 2024-09-22T00:27:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1872 2024-09-22T00:27:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1873 2024-09-22T00:27:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:27:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:27:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:27:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1874 2024-09-22T00:27:19.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1875 2024-09-22T00:27:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:27:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:27:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:27:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:27:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:27:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:27:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:27:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:27:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:27:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1876 2024-09-22T00:27:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:27:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:27:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:27:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:27:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:27:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:27:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1878 2024-09-22T00:27:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1879 2024-09-22T00:27:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1877 2024-09-22T00:27:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:27:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:27:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:27:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:27:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:27:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:27:21.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:27:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:27:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:27:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:27:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:27:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:27:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1880 2024-09-22T00:27:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1881 2024-09-22T00:27:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:27:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:27:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:27:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1882 2024-09-22T00:27:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1883 2024-09-22T00:27:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1884 2024-09-22T00:27:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1885 2024-09-22T00:27:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:27:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:27:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:27:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:27:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:27:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:27:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:27:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:27:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:27:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1886 2024-09-22T00:27:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:27:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:27:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:27:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:27:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:27:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:27:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1889 2024-09-22T00:27:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1887 2024-09-22T00:27:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:27:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:27:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:27:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1888 2024-09-22T00:27:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:27:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:27:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:27:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:27:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:27:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:27:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:27:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:27:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:27:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1890 2024-09-22T00:27:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1891 2024-09-22T00:27:27.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1892 2024-09-22T00:27:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:27:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:27:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:27:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1893 2024-09-22T00:27:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1894 2024-09-22T00:27:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1895 2024-09-22T00:27:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:27:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:27:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:27:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:27:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:27:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:27:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:27:29.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:29.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:29.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:27:29.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:27:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1896 2024-09-22T00:27:30.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:27:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:27:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:27:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:27:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:27:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:27:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1897 2024-09-22T00:27:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1899 2024-09-22T00:27:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1898 2024-09-22T00:27:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:31.403 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:31.400+0000 7f84953c5640 -1 reset not still connected to 0x5636f85f0a90 2024-09-22T00:27:31.403 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:31.400+0000 7f84953c5640 -1 reset not still connected to 0x5636f85f0d00 2024-09-22T00:27:31.404 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:31.400+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b32ea0 2024-09-22T00:27:31.404 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:31.400+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b332b0 2024-09-22T00:27:31.405 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:31.400+0000 7f84953c5640 -1 reset not still connected to 0x5636f8ec80d0 2024-09-22T00:27:31.406 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:31.404+0000 7f84953c5640 -1 reset not still connected to 0x5636f9c289c0 2024-09-22T00:27:31.406 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:31.404+0000 7f84953c5640 -1 reset not still connected to 0x5636fa732c30 2024-09-22T00:27:31.421 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:31.416+0000 7f84953c5640 -1 reset not still connected to 0x5636fd2c6750 2024-09-22T00:27:31.421 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:31.416+0000 7f84953c5640 -1 reset not still connected to 0x5637038e4820 2024-09-22T00:27:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:27:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:27:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:27:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:27:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:27:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:27:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1901 2024-09-22T00:27:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1900 2024-09-22T00:27:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:27:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:27:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:27:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:27:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:27:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:27:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1902 2024-09-22T00:27:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1903 2024-09-22T00:27:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:27:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:27:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:27:33.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1904 2024-09-22T00:27:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1905 2024-09-22T00:27:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:27:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:27:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:27:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:27:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:27:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:27:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:27:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:27:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:27:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1906 2024-09-22T00:27:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:27:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:27:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:27:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:27:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:27:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:27:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1907 2024-09-22T00:27:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1909 2024-09-22T00:27:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1908 2024-09-22T00:27:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:27:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:27:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:27:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:36.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:27:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:27:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:27:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1911 2024-09-22T00:27:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:27:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:27:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:27:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:27:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:27:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:27:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1910 2024-09-22T00:27:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:37.487 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:27:37.484+0000 7fab5fe19640 -1 reset not still connected to 0x562416bfea90 2024-09-22T00:27:37.487 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:27:37.484+0000 7fab5fe19640 -1 reset not still connected to 0x56242475c750 2024-09-22T00:27:38.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:38.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:27:38.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:38.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:38.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:27:38.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:27:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1912 2024-09-22T00:27:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1913 2024-09-22T00:27:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1915 2024-09-22T00:27:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1914 2024-09-22T00:27:39.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:27:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:27:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:27:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:27:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:27:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:27:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:27:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:27:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:27:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1916 2024-09-22T00:27:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:27:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:27:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:27:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:27:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:27:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:27:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1917 2024-09-22T00:27:41.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1919 2024-09-22T00:27:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1918 2024-09-22T00:27:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:27:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:27:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:27:41.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:41.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:27:41.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:27:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:27:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:27:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:27:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:27:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:27:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:27:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:27:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1921 2024-09-22T00:27:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1920 2024-09-22T00:27:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:42.456 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:27:42.456+0000 7fab5fe19640 -1 reset not still connected to 0x562416bfea90 2024-09-22T00:27:42.540 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:27:42.536+0000 7fab5fe19640 -1 reset not still connected to 0x5623fd831110 2024-09-22T00:27:42.600 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:27:42.536+0000 7fab5fe19640 -1 reset not still connected to 0x56240a858dd0 2024-09-22T00:27:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1922 2024-09-22T00:27:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1923 2024-09-22T00:27:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:43.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:27:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:27:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:27:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1925 2024-09-22T00:27:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1924 2024-09-22T00:27:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:27:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:27:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:27:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:27:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:27:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:27:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:27:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:27:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:27:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1926 2024-09-22T00:27:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:27:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:27:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:27:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:27:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:27:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:27:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1928 2024-09-22T00:27:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1929 2024-09-22T00:27:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1927 2024-09-22T00:27:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:27:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:27:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:27:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:27:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:27:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:27:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:27:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:27:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:27:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:27:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:27:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:27:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1931 2024-09-22T00:27:47.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1930 2024-09-22T00:27:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1932 2024-09-22T00:27:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1933 2024-09-22T00:27:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:27:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:27:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:27:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1934 2024-09-22T00:27:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1935 2024-09-22T00:27:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:27:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:27:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:27:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:27:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:27:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:27:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:27:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:27:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:27:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1936 2024-09-22T00:27:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:27:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:27:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:27:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:27:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:27:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:27:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1939 2024-09-22T00:27:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:27:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:27:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:27:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:27:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:27:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:27:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:27:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:27:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:27:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:27:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:27:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:27:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1938 2024-09-22T00:27:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1937 2024-09-22T00:27:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1940 2024-09-22T00:27:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:52.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:52.590 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:27:52.588+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7c694d380 2024-09-22T00:27:52.590 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:27:52.588+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7d0a6fad0 2024-09-22T00:27:52.590 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:27:52.588+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7d991ab60 2024-09-22T00:27:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1941 2024-09-22T00:27:53.445 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:27:53.432+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7d934fee0 2024-09-22T00:27:53.448 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:27:53.440+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7e07cc680 2024-09-22T00:27:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1943 2024-09-22T00:27:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:27:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:27:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:27:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:53.522 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:27:53.520+0000 7fab5fe19640 -1 reset not still connected to 0x5623fd831110 2024-09-22T00:27:53.522 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:27:53.520+0000 7fab5fe19640 -1 reset not still connected to 0x56240a858dd0 2024-09-22T00:27:53.527 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:27:53.524+0000 7fab5fe19640 -1 reset not still connected to 0x56241af7ed00 2024-09-22T00:27:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1945 2024-09-22T00:27:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1944 2024-09-22T00:27:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1942 2024-09-22T00:27:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:27:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:27:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:27:55.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:55.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:27:55.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:55.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:55.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:27:55.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:27:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:27:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:27:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:27:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:27:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:27:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:27:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1947 2024-09-22T00:27:56.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1946 2024-09-22T00:27:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:27:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:27:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:27:56.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:56.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:27:56.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:56.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:56.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:27:56.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:27:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1948 2024-09-22T00:27:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1949 2024-09-22T00:27:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:27:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:27:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:27:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:27:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:27:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:27:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:27:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:27:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:27:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1950 2024-09-22T00:27:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1952 2024-09-22T00:27:57.585 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:27:57.584+0000 7fab5fe19640 -1 reset not still connected to 0x5623fd831110 2024-09-22T00:27:57.585 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:27:57.584+0000 7fab5fe19640 -1 reset not still connected to 0x56240a858dd0 2024-09-22T00:27:57.590 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:27:57.588+0000 7fab5fe19640 -1 reset not still connected to 0x56241af7ed00 2024-09-22T00:27:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1951 2024-09-22T00:27:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1955 2024-09-22T00:27:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1953 2024-09-22T00:27:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1954 2024-09-22T00:27:59.323 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:59.311+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b332b0 2024-09-22T00:27:59.324 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:59.311+0000 7f84953c5640 -1 reset not still connected to 0x5636f8ec80d0 2024-09-22T00:27:59.324 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:59.311+0000 7f84953c5640 -1 reset not still connected to 0x5636f9c289c0 2024-09-22T00:27:59.324 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:59.311+0000 7f84953c5640 -1 reset not still connected to 0x5636fa732c30 2024-09-22T00:27:59.327 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:59.315+0000 7f84953c5640 -1 reset not still connected to 0x5636fbd20750 2024-09-22T00:27:59.328 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:59.315+0000 7f84953c5640 -1 reset not still connected to 0x5636fd2c6750 2024-09-22T00:27:59.328 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:27:59.315+0000 7f84953c5640 -1 reset not still connected to 0x5637038e4820 2024-09-22T00:27:59.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:27:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:27:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:27:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:27:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:27:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:27:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:27:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:27:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:27:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:27:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:27:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:27:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:27:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:27:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:27:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:28:00.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1956 2024-09-22T00:28:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:28:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:28:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:28:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:28:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:28:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:28:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:28:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:28:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 86811 2024-09-22T00:28:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1959 2024-09-22T00:28:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1958 2024-09-22T00:28:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:28:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 86938 2024-09-22T00:28:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:28:01.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:28:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:28:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:28:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:28:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:28:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:28:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:28:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:28:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:28:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:28:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:28:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:28:02.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:28:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:28:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 87065 2024-09-22T00:28:02.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1960 2024-09-22T00:28:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1957 2024-09-22T00:28:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:28:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:28:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1961 2024-09-22T00:28:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:28:03.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:28:03.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 87212 2024-09-22T00:28:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1962 2024-09-22T00:28:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:28:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:28:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:28:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:28:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:28:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:28:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:28:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:28:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:28:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:28:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:28:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:28:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:28:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:28:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 87359 2024-09-22T00:28:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:28:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 87507 2024-09-22T00:28:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1964 2024-09-22T00:28:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1963 2024-09-22T00:28:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:28:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 87655 2024-09-22T00:28:06.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:28:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:28:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:28:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 87803 2024-09-22T00:28:09.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:28:09.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 87970 2024-09-22T00:28:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:28:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:28:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 86685 2024-09-22T00:30:20.476 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:30:20.470+0000 7f84953c5640 -1 reset not still connected to 0x5636f85f0410 2024-09-22T00:30:20.476 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:30:20.470+0000 7f84953c5640 -1 reset not still connected to 0x5636f85f0750 2024-09-22T00:30:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:30:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:30:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:30:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 86812 2024-09-22T00:30:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:30:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:30:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:30:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 86939 2024-09-22T00:30:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:30:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:30:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:30:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 87066 2024-09-22T00:30:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:30:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:30:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:30:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 87213 2024-09-22T00:30:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:30:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:30:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:30:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 87360 2024-09-22T00:30:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:30:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:30:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:30:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 87508 2024-09-22T00:30:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:30:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:30:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:30:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 87656 2024-09-22T00:30:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:30:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:30:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:30:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 87804 2024-09-22T00:30:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:30:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:30:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:30:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 87971 2024-09-22T00:30:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:30:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:30:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:30:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:30:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:30:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:30:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 1 2024-09-22T00:30:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:30:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:30:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 2 2024-09-22T00:30:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd0 2024-09-22T00:30:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary1 2024-09-22T00:30:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary1 2024-09-22T00:30:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:30:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary1 2024-09-22T00:30:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd1 2024-09-22T00:30:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary2 2024-09-22T00:30:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary2 2024-09-22T00:30:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:30:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary2 2024-09-22T00:30:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:30:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:30:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 3 2024-09-22T00:30:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd2 2024-09-22T00:30:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:30:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 4 2024-09-22T00:30:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary3 2024-09-22T00:30:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary3 2024-09-22T00:30:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:30:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:30:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary3 2024-09-22T00:30:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd3 2024-09-22T00:30:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary4 2024-09-22T00:30:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary4 2024-09-22T00:30:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:30:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary4 2024-09-22T00:30:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:30:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 5 2024-09-22T00:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd4 2024-09-22T00:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary5 2024-09-22T00:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary5 2024-09-22T00:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary5 2024-09-22T00:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 6 2024-09-22T00:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd5 2024-09-22T00:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 7 2024-09-22T00:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary6 2024-09-22T00:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary6 2024-09-22T00:30:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd6 2024-09-22T00:30:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:30:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary7 2024-09-22T00:30:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary7 2024-09-22T00:30:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:30:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:30:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary7 2024-09-22T00:30:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:30:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary6 2024-09-22T00:30:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:30:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:30:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:30:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:30:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 94219 2024-09-22T00:30:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 10 2024-09-22T00:30:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd9 2024-09-22T00:30:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary10 2024-09-22T00:30:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary10 2024-09-22T00:30:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:30:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary10 2024-09-22T00:30:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 9 2024-09-22T00:30:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd8 2024-09-22T00:30:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary9 2024-09-22T00:30:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary9 2024-09-22T00:30:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:30:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary9 2024-09-22T00:30:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 8 2024-09-22T00:30:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd7 2024-09-22T00:30:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary8 2024-09-22T00:30:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary8 2024-09-22T00:30:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:30:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary8 2024-09-22T00:30:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-22T00:30:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:30:41.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-22T00:30:41.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:30:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-22T00:30:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:30:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-22T00:30:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:30:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-22T00:30:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:30:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-22T00:30:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:30:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-22T00:30:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:30:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-22T00:30:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:30:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-22T00:30:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:30:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-22T00:30:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:31:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=110b430501719f7ec69dc13c0beeab77 2024-09-22T00:31:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-22T00:31:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-22T00:31:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:31:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-22T00:31:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=41c8d8f3809473709a196e3d13116336 2024-09-22T00:31:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-22T00:31:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=966cb9a5e608d4d47accc66318d5bd8f 2024-09-22T00:31:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-22T00:31:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-22T00:31:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:31:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-22T00:31:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-22T00:31:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:31:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-22T00:31:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=4bae802ea9a9d77d3b1a48f568beccf6 2024-09-22T00:31:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-22T00:31:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=3789de56455864c57bca0ef332759518 2024-09-22T00:31:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-22T00:31:14.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-22T00:31:14.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:14.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:14.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:31:14.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-22T00:31:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-22T00:31:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:31:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-22T00:31:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=2c02470ee915b5b5a73921c2af0a92a6 2024-09-22T00:31:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-22T00:31:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-22T00:31:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:31:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-22T00:31:15.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=b4af33d8bea221878adc203834c8b949 2024-09-22T00:31:15.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-22T00:31:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=ed3b56f85a3ca9ceea961548d3a8156d 2024-09-22T00:31:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-22T00:31:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-22T00:31:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:31:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-22T00:31:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=827354d15776ba321ac02a99a35db543 2024-09-22T00:31:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-22T00:31:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-22T00:31:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:15.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:15.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:31:15.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-22T00:31:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-22T00:31:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:31:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-22T00:31:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=7b793ce829e7c043e858d45525048f22 2024-09-22T00:31:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-22T00:31:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:31:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-22T00:31:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:31:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-22T00:31:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-22T00:31:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:31:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:31:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-22T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:15.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:15.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:15.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:31:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-22T00:31:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:31:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-22T00:31:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:31:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-22T00:31:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:31:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:31:16.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-22T00:31:16.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:16.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:16.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:31:16.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:16.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:16.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:16.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:16.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:16.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-22T00:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:31:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-22T00:31:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:31:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:31:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:31:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 39691262-dc09-459e-a706-083041ddeb1b 2024-09-22T00:31:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:15 2024-09-22T00:31:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:31:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:16' 2024-09-22T00:31:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:16.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:31:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e11fe15e-b1f5-4761-b5de-fa0922f20c54 2024-09-22T00:31:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:15 2024-09-22T00:31:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:31:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:16' 2024-09-22T00:31:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:31:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:31:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:31:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-22T00:31:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:31:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:31:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:15' 2024-09-22T00:31:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-22 00:31:15' 2024-09-22T00:31:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:31:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:15' 2024-09-22T00:31:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:31:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:31:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6d02ae97-ffa3-463b-bb5f-e1ee89d8d122 2024-09-22T00:31:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:15 2024-09-22T00:31:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:31:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:16' 2024-09-22T00:31:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:15' 2024-09-22T00:31:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-22 00:31:15' 2024-09-22T00:31:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:31:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:15' 2024-09-22T00:31:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-22T00:31:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:31:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:31:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:31:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8da3684b-bb34-4654-9135-6980fd409d92 2024-09-22T00:31:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:31:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:31:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:15 2024-09-22T00:31:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:31:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1701888.0,"last_snapshot_bytes":1675264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964880,"remote_snapshot_timestamp":1726964880,"replay_state":"idle"} 2024-09-22T00:31:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:05' 2024-09-22T00:31:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:31:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:15' 2024-09-22T00:31:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:31:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-22 00:31:15' 2024-09-22T00:31:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:31:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:15' 2024-09-22T00:31:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-22T00:31:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:31:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING3742614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1728512.0,"last_snapshot_bytes":1732608,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964877,"remote_snapshot_timestamp":1726964877,"replay_state":"idle"}2024-09-22 00:31:05image-primary1039691262-dc09-459e-a706-083041ddeb1bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:31:16image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1675264.0,"last_snapshot_bytes":1609728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964877,"remote_snapshot_timestamp":1726964877,"replay_state":"idle"}2024-09-22 00:31:05image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1724416.0,"last_snapshot_bytes":1679360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964878,"remote_snapshot_timestamp":1726964878,"replay_state":"idle"}2024-09-22 00:31:05image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2914304.0,"last_snapshot_bytes":4014080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964880,"remote_snapshot_timestamp":1726964880,"replay_state":"idle"}2024-09-22 00:31:05image-primary58da3684b-bb34-4654-9135-6980fd409d92up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1701888.0,"last_snapshot_bytes":1675264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964880,"remote_snapshot_timestamp":1726964880,"replay_state":"idle"}2024-09-22 00:31:05image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1714176.0,"last_snapshot_bytes":1597440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964881,"remote_snapshot_timestamp":1726964881,"replay_state":"idle"}2024-09-22 00:31:05image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1712128.0,"last_snapshot_bytes":1683456,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964882,"remote_snapshot_timestamp":1726964882,"replay_state":"idle"}2024-09-22 00:31:05image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:31:16image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:31:16' 2024-09-22T00:31:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:31:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING3742614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1728512.0,"last_snapshot_bytes":1732608,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964877,"remote_snapshot_timestamp":1726964877,"replay_state":"idle"}2024-09-22 00:31:05image-primary1039691262-dc09-459e-a706-083041ddeb1bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:31:16image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1675264.0,"last_snapshot_bytes":1609728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964877,"remote_snapshot_timestamp":1726964877,"replay_state":"idle"}2024-09-22 00:31:05image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1724416.0,"last_snapshot_bytes":1679360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964878,"remote_snapshot_timestamp":1726964878,"replay_state":"idle"}2024-09-22 00:31:05image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2914304.0,"last_snapshot_bytes":4014080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964880,"remote_snapshot_timestamp":1726964880,"replay_state":"idle"}2024-09-22 00:31:05image-primary58da3684b-bb34-4654-9135-6980fd409d92up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1701888.0,"last_snapshot_bytes":1675264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964880,"remote_snapshot_timestamp":1726964880,"replay_state":"idle"}2024-09-22 00:31:05image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1714176.0,"last_snapshot_bytes":1597440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964881,"remote_snapshot_timestamp":1726964881,"replay_state":"idle"}2024-09-22 00:31:05image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1712128.0,"last_snapshot_bytes":1683456,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964882,"remote_snapshot_timestamp":1726964882,"replay_state":"idle"}2024-09-22 00:31:05image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:31:16image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:31:16' 2024-09-22T00:31:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:31:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING3742614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1728512.0,"last_snapshot_bytes":1732608,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964877,"remote_snapshot_timestamp":1726964877,"replay_state":"idle"}2024-09-22 00:31:05image-primary1039691262-dc09-459e-a706-083041ddeb1bup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:31:16image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1675264.0,"last_snapshot_bytes":1609728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964877,"remote_snapshot_timestamp":1726964877,"replay_state":"idle"}2024-09-22 00:31:05image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1724416.0,"last_snapshot_bytes":1679360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964878,"remote_snapshot_timestamp":1726964878,"replay_state":"idle"}2024-09-22 00:31:05image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2914304.0,"last_snapshot_bytes":4014080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964880,"remote_snapshot_timestamp":1726964880,"replay_state":"idle"}2024-09-22 00:31:05image-primary58da3684b-bb34-4654-9135-6980fd409d92up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1701888.0,"last_snapshot_bytes":1675264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964880,"remote_snapshot_timestamp":1726964880,"replay_state":"idle"}2024-09-22 00:31:05image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1714176.0,"last_snapshot_bytes":1597440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964881,"remote_snapshot_timestamp":1726964881,"replay_state":"idle"}2024-09-22 00:31:05image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1712128.0,"last_snapshot_bytes":1683456,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964882,"remote_snapshot_timestamp":1726964882,"replay_state":"idle"}2024-09-22 00:31:05image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:31:16image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:31:16' 2024-09-22T00:31:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:31:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:15' 2024-09-22T00:31:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:15' 2024-09-22T00:31:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:15' 2024-09-22T00:31:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:31:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:31:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:31:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-22T00:31:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:31:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-22T00:31:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-22T00:31:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:31:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:31:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:31:17.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:31:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:31:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-22T00:31:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:31:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-22T00:31:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:31:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-22T00:31:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:31:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-22T00:31:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:31:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:31:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-22T00:31:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:31:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:31:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-22T00:31:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:31:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:31:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:31:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 3623ca88-98f0-4728-9ca6-b784db8b4075 2024-09-22T00:31:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:31:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:31:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:15 2024-09-22T00:31:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:31:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:17' 2024-09-22T00:31:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:31:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:31:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:31:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 375b2e3e-524e-40f4-a41f-19efac72a51a 2024-09-22T00:31:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:31:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:31:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:15 2024-09-22T00:31:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:31:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:17' 2024-09-22T00:31:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:31:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:31:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:31:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: b5dc0f6c-1ed1-4c7a-a2ae-86146375d79e 2024-09-22T00:31:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:31:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:31:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:15 2024-09-22T00:31:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:31:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:17' 2024-09-22T00:31:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:31:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:31:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-22T00:31:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-22T00:31:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:31:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:31:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-22T00:31:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:31:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-22T00:31:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-22T00:31:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-22T00:31:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:31:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e11fe15e-b1f5-4761-b5de-fa0922f20c54 2024-09-22T00:31:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:31:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:16 2024-09-22T00:31:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:31:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:15' 2024-09-22T00:31:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:31:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:18.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:16' 2024-09-22T00:31:18.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-22 00:31:16' 2024-09-22T00:31:18.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:31:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:16' 2024-09-22T00:31:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:31:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 39691262-dc09-459e-a706-083041ddeb1b 2024-09-22T00:31:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:31:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:16 2024-09-22T00:31:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:31:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:15' 2024-09-22T00:31:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:31:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:16' 2024-09-22T00:31:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-22 00:31:16' 2024-09-22T00:31:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:31:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:16' 2024-09-22T00:31:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:31:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6d02ae97-ffa3-463b-bb5f-e1ee89d8d122 2024-09-22T00:31:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:31:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:16 2024-09-22T00:31:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:31:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:15' 2024-09-22T00:31:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:31:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:16' 2024-09-22T00:31:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-22 00:31:16' 2024-09-22T00:31:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:18.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:31:18.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:18.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:16' 2024-09-22T00:31:18.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:18.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:18.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1728512.0,"last_snapshot_bytes":1732608,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964877,"remote_snapshot_timestamp":1726964877,"replay_state":"idle"}42634269mirror.0smithi0602024-09-22 00:31:05cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary1039691262-dc09-459e-a706-083041ddeb1bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:16cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:15image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1675264.0,"last_snapshot_bytes":1609728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964877,"remote_snapshot_timestamp":1726964877,"replay_state":"idle"}42634269mirror.0smithi0602024-09-22 00:31:05cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1724416.0,"last_snapshot_bytes":1679360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964878,"remote_snapshot_timestamp":1726964878,"replay_state":"idle"}42634269mirror.0smithi0602024-09-22 00:31:05cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:17cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary58da3684b-bb34-4654-9135-6980fd409d92up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:17cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:17cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:17cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:16cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:15image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:16cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:15' 2024-09-22T00:31:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:31:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:16' 2024-09-22T00:31:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:31:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+stopping_replayremote image demoted42634269mirror.0smithi0602024-09-22 00:31:18cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary1039691262-dc09-459e-a706-083041ddeb1bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:16cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:15image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1675264.0,"last_snapshot_bytes":1609728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726964877,"remote_snapshot_timestamp":1726964877,"replay_state":"idle"}42634269mirror.0smithi0602024-09-22 00:31:05cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+stopping_replayremote image demoted42634269mirror.0smithi0602024-09-22 00:31:18cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:17cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary58da3684b-bb34-4654-9135-6980fd409d92up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:17cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:17cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:17cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:16cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:15image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:16cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:15' 2024-09-22T00:31:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:31:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-22T00:31:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:31:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:31:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-22T00:31:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+stopping_replayremote image demoted42634269mirror.0smithi0602024-09-22 00:31:18cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary1039691262-dc09-459e-a706-083041ddeb1bup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:16cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:15image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:18cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+stopping_replayremote image demoted42634269mirror.0smithi0602024-09-22 00:31:18cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:17cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary58da3684b-bb34-4654-9135-6980fd409d92up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:17cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:17cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:17cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:31:15image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:16cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:15image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:31:16cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:15' 2024-09-22T00:31:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:31:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:16' 2024-09-22T00:31:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:31:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:16' 2024-09-22T00:31:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:31:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-22T00:31:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:31:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:31:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-22T00:31:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-22T00:31:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:31:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:31:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-22T00:31:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-22T00:31:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:31:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:18.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:18.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:31:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-22T00:31:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:31:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:31:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-22T00:31:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:31:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:31:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:31:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9e2e9109-8e5b-413c-8411-9ba5ae31be5f 2024-09-22T00:31:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:31:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:31:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:15 2024-09-22T00:31:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:31:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:18' 2024-09-22T00:31:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:31:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: b7c8b91b-78db-4a95-a282-7a3cffdfd580 2024-09-22T00:31:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:31:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:31:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:15 2024-09-22T00:31:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:31:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:18' 2024-09-22T00:31:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:31:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:31:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:31:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:31:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:31:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 75ab6c51-4a42-443e-a205-28783bbf691c 2024-09-22T00:31:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:31:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:31:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:15 2024-09-22T00:31:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:31:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:18' 2024-09-22T00:31:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:31:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:31:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-22T00:31:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:31:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:31:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:31:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8da3684b-bb34-4654-9135-6980fd409d92 2024-09-22T00:31:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:31:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:31:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:15 2024-09-22T00:31:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:31:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:17' 2024-09-22T00:31:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:31:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-09-22T00:31:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:31:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:31:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:31:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:31:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:31:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-22T00:31:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:31:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:31:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-22T00:31:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:31:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:31:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-22T00:31:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd0 2024-09-22T00:31:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-22T00:31:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:31:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd2 2024-09-22T00:31:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd1 2024-09-22T00:31:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-22T00:31:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-22T00:31:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:31:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:31:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-22T00:31:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:31:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:31:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-22T00:31:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:31:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:31:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-22T00:31:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:31:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:31:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:31:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 3623ca88-98f0-4728-9ca6-b784db8b4075 2024-09-22T00:31:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:31:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:31:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:15 2024-09-22T00:31:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:31:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:17' 2024-09-22T00:31:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:31:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-09-22T00:31:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:31:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 375b2e3e-524e-40f4-a41f-19efac72a51a 2024-09-22T00:31:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:31:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:31:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:15 2024-09-22T00:31:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:31:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:17' 2024-09-22T00:31:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:31:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-09-22T00:31:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:31:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: b5dc0f6c-1ed1-4c7a-a2ae-86146375d79e 2024-09-22T00:31:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:31:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:31:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:15 2024-09-22T00:31:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:31:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:17' 2024-09-22T00:31:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:31:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-09-22T00:31:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-22T00:31:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:31:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-22T00:31:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:31:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:31:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:31:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-22T00:31:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:31:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:31:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:31:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: b7c8b91b-78db-4a95-a282-7a3cffdfd580 2024-09-22T00:31:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20 2024-09-22T00:31:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20' 2024-09-22T00:31:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:31:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-22 00:31:20' 2024-09-22T00:31:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:31:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:20' 2024-09-22T00:31:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:31:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 75ab6c51-4a42-443e-a205-28783bbf691c 2024-09-22T00:31:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20 2024-09-22T00:31:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20' 2024-09-22T00:31:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:31:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:31:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9e2e9109-8e5b-413c-8411-9ba5ae31be5f 2024-09-22T00:31:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20 2024-09-22T00:31:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20' 2024-09-22T00:31:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:31:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-22 00:31:20' 2024-09-22T00:31:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:31:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:20' 2024-09-22T00:31:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-22 00:31:20' 2024-09-22T00:31:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:31:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:20' 2024-09-22T00:31:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary1039691262-dc09-459e-a706-083041ddeb1bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary58da3684b-bb34-4654-9135-6980fd409d92up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20' 2024-09-22T00:31:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:31:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:31:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-22T00:31:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:31:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary1039691262-dc09-459e-a706-083041ddeb1bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary58da3684b-bb34-4654-9135-6980fd409d92up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20' 2024-09-22T00:31:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:31:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:31:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-22T00:31:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:31:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary1039691262-dc09-459e-a706-083041ddeb1bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary58da3684b-bb34-4654-9135-6980fd409d92up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20' 2024-09-22T00:31:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:31:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:31:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-22T00:31:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:31:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-22T00:31:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:31:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-22T00:31:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:31:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: b7c8b91b-78db-4a95-a282-7a3cffdfd580 2024-09-22T00:31:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20 2024-09-22T00:31:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:31:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20' 2024-09-22T00:31:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:31:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:22.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-22 00:31:20' 2024-09-22T00:31:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:31:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:20' 2024-09-22T00:31:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-22T00:31:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:31:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-22T00:31:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-22T00:31:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:31:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-22T00:31:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:31:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9e2e9109-8e5b-413c-8411-9ba5ae31be5f 2024-09-22T00:31:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20 2024-09-22T00:31:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:31:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20' 2024-09-22T00:31:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:31:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:22.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:22.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-22 00:31:20' 2024-09-22T00:31:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:31:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:20' 2024-09-22T00:31:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:31:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 75ab6c51-4a42-443e-a205-28783bbf691c 2024-09-22T00:31:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20 2024-09-22T00:31:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:31:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20' 2024-09-22T00:31:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:31:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary1039691262-dc09-459e-a706-083041ddeb1bup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary58da3684b-bb34-4654-9135-6980fd409d92up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20' 2024-09-22T00:31:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:31:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-22 00:31:20' 2024-09-22T00:31:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:31:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:20' 2024-09-22T00:31:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:22.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:31:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:22.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-22T00:31:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:31:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:31:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-22T00:31:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary1039691262-dc09-459e-a706-083041ddeb1bup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary58da3684b-bb34-4654-9135-6980fd409d92up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20' 2024-09-22T00:31:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:31:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:31:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-22T00:31:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:31:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:31:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-22T00:31:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary1039691262-dc09-459e-a706-083041ddeb1bup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary58da3684b-bb34-4654-9135-6980fd409d92up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20' 2024-09-22T00:31:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:31:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:31:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-22T00:31:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:31:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:31:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-22T00:31:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-22T00:31:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:31:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:31:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:31:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8da3684b-bb34-4654-9135-6980fd409d92 2024-09-22T00:31:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20 2024-09-22T00:31:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20' 2024-09-22T00:31:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:31:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-22 00:31:20' 2024-09-22T00:31:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:31:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:20' 2024-09-22T00:31:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:31:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:31:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:31:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:31:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:31:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-22T00:31:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:31:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:31:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-22T00:31:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:31:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:31:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-22T00:31:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary1039691262-dc09-459e-a706-083041ddeb1bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary58da3684b-bb34-4654-9135-6980fd409d92up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20' 2024-09-22T00:31:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:31:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:31:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-22T00:31:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:31:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd3 2024-09-22T00:31:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-22T00:31:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:31:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd4 2024-09-22T00:31:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:31:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-22T00:31:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd5 2024-09-22T00:31:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:31:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-22T00:31:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-22T00:31:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:31:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:31:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-22T00:31:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:31:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:31:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-22T00:31:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:31:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:31:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:31:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 3623ca88-98f0-4728-9ca6-b784db8b4075 2024-09-22T00:31:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20 2024-09-22T00:31:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20' 2024-09-22T00:31:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:31:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-22 00:31:20' 2024-09-22T00:31:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:31:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:20' 2024-09-22T00:31:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:31:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 375b2e3e-524e-40f4-a41f-19efac72a51a 2024-09-22T00:31:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20 2024-09-22T00:31:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20' 2024-09-22T00:31:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:31:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: b5dc0f6c-1ed1-4c7a-a2ae-86146375d79e 2024-09-22T00:31:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20 2024-09-22T00:31:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20' 2024-09-22T00:31:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:31:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:31:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-22 00:31:20' 2024-09-22T00:31:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:31:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:20' 2024-09-22T00:31:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-22 00:31:20' 2024-09-22T00:31:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:24.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:31:24.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:24.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:20' 2024-09-22T00:31:24.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:24.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:24.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-22T00:31:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:31:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-22T00:31:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary1039691262-dc09-459e-a706-083041ddeb1bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary58da3684b-bb34-4654-9135-6980fd409d92up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20' 2024-09-22T00:31:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:31:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:31:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:24.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:24.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-22T00:31:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:31:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:31:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8da3684b-bb34-4654-9135-6980fd409d92 2024-09-22T00:31:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20 2024-09-22T00:31:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:31:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:20' 2024-09-22T00:31:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:31:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-22 00:31:20' 2024-09-22T00:31:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:31:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:20' 2024-09-22T00:31:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary1039691262-dc09-459e-a706-083041ddeb1bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary58da3684b-bb34-4654-9135-6980fd409d92up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20' 2024-09-22T00:31:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:31:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:31:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary1039691262-dc09-459e-a706-083041ddeb1bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary58da3684b-bb34-4654-9135-6980fd409d92up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:31:20image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:20cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:20' 2024-09-22T00:31:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-22T00:31:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:31:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:31:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:31:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-22T00:31:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:31:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary1039691262-dc09-459e-a706-083041ddeb1bup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary58da3684b-bb34-4654-9135-6980fd409d92up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:20image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20' 2024-09-22T00:31:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:31:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:20' 2024-09-22T00:31:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:31:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-22T00:31:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:31:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:31:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-22T00:31:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-22T00:31:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:31:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-22T00:31:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-22T00:31:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:31:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-22T00:31:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-22T00:31:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:31:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-22T00:31:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:31:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:31:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:31:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-22T00:31:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:31:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 3623ca88-98f0-4728-9ca6-b784db8b4075 2024-09-22T00:31:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:25 2024-09-22T00:31:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:31:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:25' 2024-09-22T00:31:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:31:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:25.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:31:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: b5dc0f6c-1ed1-4c7a-a2ae-86146375d79e 2024-09-22T00:31:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:25 2024-09-22T00:31:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:31:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:25' 2024-09-22T00:31:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:31:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:25' 2024-09-22T00:31:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-22 00:31:25' 2024-09-22T00:31:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:31:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:25' 2024-09-22T00:31:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:31:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 375b2e3e-524e-40f4-a41f-19efac72a51a 2024-09-22T00:31:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:25 2024-09-22T00:31:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:31:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:31:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:31:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:25' 2024-09-22T00:31:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:31:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:25' 2024-09-22T00:31:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-22 00:31:25' 2024-09-22T00:31:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:31:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:25' 2024-09-22T00:31:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:31:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:31:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:25.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:25' 2024-09-22T00:31:25.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-22 00:31:25' 2024-09-22T00:31:25.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:25.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:25.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:31:25.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:31:25.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:25' 2024-09-22T00:31:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd6 2024-09-22T00:31:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-22T00:31:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:31:25.919 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:31:25.917+0000 7fab5fe19640 -1 reset not still connected to 0x562407cc4410 2024-09-22T00:31:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING4642634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"}2024-09-22 00:31:25image-primary1039691262-dc09-459e-a706-083041ddeb1bup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"}2024-09-22 00:31:25image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"}2024-09-22 00:31:25image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:25image-primary58da3684b-bb34-4654-9135-6980fd409d92up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:25image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:25image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:25image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20' 2024-09-22T00:31:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:31:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:25' 2024-09-22T00:31:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:31:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-22T00:31:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:31:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:31:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-22T00:31:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING4642634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"}2024-09-22 00:31:25image-primary1039691262-dc09-459e-a706-083041ddeb1bup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"}2024-09-22 00:31:25image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"}2024-09-22 00:31:25image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:25image-primary58da3684b-bb34-4654-9135-6980fd409d92up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:25image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:25image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:25image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20' 2024-09-22T00:31:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:26.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:31:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING4642634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"}2024-09-22 00:31:25image-primary1039691262-dc09-459e-a706-083041ddeb1bup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"}2024-09-22 00:31:25image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"}2024-09-22 00:31:25image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:25image-primary58da3684b-bb34-4654-9135-6980fd409d92up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:25image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:25image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:31:25image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:31:25cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}2024-09-22 00:31:20' 2024-09-22T00:31:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:31:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:25' 2024-09-22T00:31:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:31:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:25' 2024-09-22T00:31:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-22T00:31:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:31:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:31:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-22T00:31:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:31:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:31:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:31:26.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:31:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:31:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-22T00:31:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:31:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:31:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-22T00:31:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:31:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:31:26.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:31:26.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:31:26.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:31:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-22T00:31:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:31:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:31:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-22T00:31:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:31:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:31:26.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-22T00:31:26.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd7 2024-09-22T00:31:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-22T00:31:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:31:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd8 2024-09-22T00:31:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:31:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-22T00:31:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd9 2024-09-22T00:31:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-22T00:31:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:31:43.440 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:31:43.433+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a37217c270 2024-09-22T00:31:43.440 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:31:43.433+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a37a9c92b0 2024-09-22T00:31:43.440 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:31:43.433+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a37e672340 2024-09-22T00:31:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=110b430501719f7ec69dc13c0beeab77 2024-09-22T00:31:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-22T00:31:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 110b430501719f7ec69dc13c0beeab77 != \1\1\0\b\4\3\0\5\0\1\7\1\9\f\7\e\c\6\9\d\c\1\3\c\0\b\e\e\a\b\7\7 ]] 2024-09-22T00:31:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=41c8d8f3809473709a196e3d13116336 2024-09-22T00:31:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-22T00:31:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 41c8d8f3809473709a196e3d13116336 != \4\1\c\8\d\8\f\3\8\0\9\4\7\3\7\0\9\a\1\9\6\e\3\d\1\3\1\1\6\3\3\6 ]] 2024-09-22T00:31:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=966cb9a5e608d4d47accc66318d5bd8f 2024-09-22T00:31:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-22T00:31:51.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 966cb9a5e608d4d47accc66318d5bd8f != \9\6\6\c\b\9\a\5\e\6\0\8\d\4\d\4\7\a\c\c\c\6\6\3\1\8\d\5\b\d\8\f ]] 2024-09-22T00:31:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=7b793ce829e7c043e858d45525048f22 2024-09-22T00:31:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-22T00:31:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 7b793ce829e7c043e858d45525048f22 != \7\b\7\9\3\c\e\8\2\9\e\7\c\0\4\3\e\8\5\8\d\4\5\5\2\5\0\4\8\f\2\2 ]] 2024-09-22T00:31:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:31:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 94220 2024-09-22T00:31:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=ed3b56f85a3ca9ceea961548d3a8156d 2024-09-22T00:31:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-22T00:31:55.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ ed3b56f85a3ca9ceea961548d3a8156d != \e\d\3\b\5\6\f\8\5\a\3\c\a\9\c\e\e\a\9\6\1\5\4\8\d\3\a\8\1\5\6\d ]] 2024-09-22T00:31:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=827354d15776ba321ac02a99a35db543 2024-09-22T00:31:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-22T00:31:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 827354d15776ba321ac02a99a35db543 != \8\2\7\3\5\4\d\1\5\7\7\6\b\a\3\2\1\a\c\0\2\a\9\9\a\3\5\d\b\5\4\3 ]] 2024-09-22T00:31:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:31:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 94221 2024-09-22T00:31:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:31:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 94224 2024-09-22T00:31:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=2c02470ee915b5b5a73921c2af0a92a6 2024-09-22T00:31:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-22T00:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 2c02470ee915b5b5a73921c2af0a92a6 != \2\c\0\2\4\7\0\e\e\9\1\5\b\5\b\5\a\7\3\9\2\1\c\2\a\f\0\a\9\2\a\6 ]] 2024-09-22T00:31:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=b4af33d8bea221878adc203834c8b949 2024-09-22T00:31:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-22T00:31:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ b4af33d8bea221878adc203834c8b949 != \b\4\a\f\3\3\d\8\b\e\a\2\2\1\8\7\8\a\d\c\2\0\3\8\3\4\c\8\b\9\4\9 ]] 2024-09-22T00:31:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=4bae802ea9a9d77d3b1a48f568beccf6 2024-09-22T00:31:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-22T00:31:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 4bae802ea9a9d77d3b1a48f568beccf6 != \4\b\a\e\8\0\2\e\a\9\a\9\d\7\7\d\3\b\1\a\4\8\f\5\6\8\b\e\c\c\f\6 ]] 2024-09-22T00:31:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=3789de56455864c57bca0ef332759518 2024-09-22T00:31:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-22T00:31:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 3789de56455864c57bca0ef332759518 != \3\7\8\9\d\e\5\6\4\5\5\8\6\4\c\5\7\b\c\a\0\e\f\3\3\2\7\5\9\5\1\8 ]] 2024-09-22T00:31:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:31:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 94225 2024-09-22T00:31:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:31:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 94228 2024-09-22T00:31:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:31:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 94230 2024-09-22T00:31:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:31:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 94231 2024-09-22T00:31:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:31:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 94232 2024-09-22T00:31:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:31:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 94234 2024-09-22T00:31:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:31:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:31:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-22T00:31:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-22T00:31:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:31:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:31:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-22T00:31:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:31:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:31:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:31:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:31:59.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-22T00:31:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:31:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:31:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:31:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:31:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:31:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:31:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9e2e9109-8e5b-413c-8411-9ba5ae31be5f 2024-09-22T00:31:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:31:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":42168320.0,"bytes_per_snapshot":72114176.0,"last_snapshot_bytes":72114176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965083,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"} 2024-09-22T00:31:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:31:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:35 2024-09-22T00:31:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:31:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:31:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:31:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:31:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:55' 2024-09-22T00:31:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:31:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:31:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:31:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":42168320.0,"bytes_per_snapshot":72114176.0,"last_snapshot_bytes":72114176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965083,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"} 2024-09-22T00:31:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:31:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:31:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:31:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:31:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:31:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:31:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:31:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:31:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:35' 2024-09-22T00:31:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-22 00:31:35' 2024-09-22T00:31:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:31:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:31:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:31:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:35' 2024-09-22T00:31:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:31:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:31:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:31:59.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary19e2e9109-8e5b-413c-8411-9ba5ae31be5fup+replayingreplaying, {"bytes_per_second":42168320.0,"bytes_per_snapshot":72114176.0,"last_snapshot_bytes":72114176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965083,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:55image-primary1039691262-dc09-459e-a706-083041ddeb1bup+replayingreplaying, {"bytes_per_second":6500352.0,"bytes_per_snapshot":65003520.0,"last_snapshot_bytes":65003520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:55image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+replayingreplaying, {"bytes_per_second":55705600.0,"bytes_per_snapshot":72224768.0,"last_snapshot_bytes":72224768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965083,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:55image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+replayingreplaying, {"bytes_per_second":40042496.0,"bytes_per_snapshot":71843840.0,"last_snapshot_bytes":71843840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965083,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:55image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+replayingreplaying, {"bytes_per_second":46870528.0,"bytes_per_snapshot":70012928.0,"last_snapshot_bytes":70012928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:55image-primary58da3684b-bb34-4654-9135-6980fd409d92up+replayingreplaying, {"bytes_per_second":14680064.0,"bytes_per_snapshot":71946240.0,"last_snapshot_bytes":71946240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965085,"remote_snapshot_timestamp":1726965085,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:55image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+replayingreplaying, {"bytes_per_second":37588992.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:55image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+replayingreplaying, {"bytes_per_second":16011264.0,"bytes_per_snapshot":71802880.0,"last_snapshot_bytes":71802880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:55image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+replayingreplaying, {"bytes_per_second":7228620.8,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:55image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+replayingreplaying, {"bytes_per_second":6691635.2,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:31:55' 2024-09-22T00:31:59.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:31:59.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:31:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:35' 2024-09-22T00:31:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:31:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:31:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:31:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:31:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:31:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:31:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-22T00:31:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:31:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:31:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:31:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-22T00:31:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-22T00:32:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:32:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-22T00:32:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:32:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:32:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary1 2024-09-22T00:32:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary1 2024-09-22T00:32:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-22T00:32:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-22T00:32:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:32:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:32:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-22T00:32:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:32:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:32:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:32:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:32:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-22T00:32:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:32:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:32:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:32:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:32:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:32:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:32:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:32:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: b7c8b91b-78db-4a95-a282-7a3cffdfd580 2024-09-22T00:32:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:32:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":55705600.0,"bytes_per_snapshot":72224768.0,"last_snapshot_bytes":72224768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965083,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"} 2024-09-22T00:32:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:32:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:31:35 2024-09-22T00:32:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:32:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:32:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:32:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:32:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:32:00' 2024-09-22T00:32:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:32:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:32:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:32:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":55705600.0,"bytes_per_snapshot":72224768.0,"last_snapshot_bytes":72224768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965083,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"} 2024-09-22T00:32:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:32:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:32:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:32:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:32:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:32:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:32:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:32:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:32:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:35' 2024-09-22T00:32:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-22 00:31:35' 2024-09-22T00:32:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:32:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:31:35' 2024-09-22T00:32:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:32:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:32:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:32:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK942614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1039691262-dc09-459e-a706-083041ddeb1bup+replayingreplaying, {"bytes_per_second":6500352.0,"bytes_per_snapshot":65003520.0,"last_snapshot_bytes":65003520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:00image-primary2b7c8b91b-78db-4a95-a282-7a3cffdfd580up+replayingreplaying, {"bytes_per_second":55705600.0,"bytes_per_snapshot":72224768.0,"last_snapshot_bytes":72224768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965083,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:00image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+replayingreplaying, {"bytes_per_second":40042496.0,"bytes_per_snapshot":71843840.0,"last_snapshot_bytes":71843840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965083,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:00image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+replayingreplaying, {"bytes_per_second":46870528.0,"bytes_per_snapshot":70012928.0,"last_snapshot_bytes":70012928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:00image-primary58da3684b-bb34-4654-9135-6980fd409d92up+replayingreplaying, {"bytes_per_second":14680064.0,"bytes_per_snapshot":71946240.0,"last_snapshot_bytes":71946240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965085,"remote_snapshot_timestamp":1726965085,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:00image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+replayingreplaying, {"bytes_per_second":37588992.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:00image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+replayingreplaying, {"bytes_per_second":16011264.0,"bytes_per_snapshot":71802880.0,"last_snapshot_bytes":71802880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:00image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+replayingreplaying, {"bytes_per_second":7228620.8,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:00image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+replayingreplaying, {"bytes_per_second":6691635.2,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:31:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:00' 2024-09-22T00:32:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:32:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:32:02.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:31:35' 2024-09-22T00:32:02.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:32:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:32:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:32:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:32:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:32:02.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:02.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:02.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-22T00:32:02.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:32:02.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:02.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:32:02.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-22T00:32:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-22T00:32:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:32:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-22T00:32:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:32:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:32:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary2 2024-09-22T00:32:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary2 2024-09-22T00:32:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-22T00:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-22T00:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-22T00:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:32:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-22T00:32:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:32:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:32:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:32:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:32:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:32:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:32:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:32:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 75ab6c51-4a42-443e-a205-28783bbf691c 2024-09-22T00:32:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:32:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2394794.67,"bytes_per_snapshot":71843840.0,"last_snapshot_bytes":71843840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965083,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"} 2024-09-22T00:32:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:32:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:32:05 2024-09-22T00:32:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:32:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:32:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:32:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:32:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:32:05' 2024-09-22T00:32:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:32:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:32:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:32:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2394794.67,"bytes_per_snapshot":71843840.0,"last_snapshot_bytes":71843840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965083,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"} 2024-09-22T00:32:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:32:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:32:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:32:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:32:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:32:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:32:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:32:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:32:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:32:05' 2024-09-22T00:32:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-22 00:32:05' 2024-09-22T00:32:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:32:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:32:05' 2024-09-22T00:32:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:32:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:32:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:32:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK842614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1039691262-dc09-459e-a706-083041ddeb1bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65003520.0,"last_snapshot_bytes":65003520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:05image-primary375ab6c51-4a42-443e-a205-28783bbf691cup+replayingreplaying, {"bytes_per_second":2394794.67,"bytes_per_snapshot":71843840.0,"last_snapshot_bytes":71843840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965083,"remote_snapshot_timestamp":1726965083,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:05image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+replayingreplaying, {"bytes_per_second":2333764.27,"bytes_per_snapshot":70012928.0,"last_snapshot_bytes":70012928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:05image-primary58da3684b-bb34-4654-9135-6980fd409d92up+replayingreplaying, {"bytes_per_second":2398208.0,"bytes_per_snapshot":71946240.0,"last_snapshot_bytes":71946240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965085,"remote_snapshot_timestamp":1726965085,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:05image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+replayingreplaying, {"bytes_per_second":2406400.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:05image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+replayingreplaying, {"bytes_per_second":2393429.33,"bytes_per_snapshot":71802880.0,"last_snapshot_bytes":71802880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:05image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:05image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:05' 2024-09-22T00:32:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:32:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:32:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:32:05' 2024-09-22T00:32:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:32:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:32:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:32:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:32:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:32:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-22T00:32:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:32:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:32:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-22T00:32:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-22T00:32:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:32:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-22T00:32:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:32:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:32:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary3 2024-09-22T00:32:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:07.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary3 2024-09-22T00:32:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-22T00:32:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-22T00:32:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:32:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:32:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-22T00:32:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:32:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:32:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:32:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:32:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-22T00:32:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:32:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:32:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:32:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:32:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:32:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:32:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:32:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: b5dc0f6c-1ed1-4c7a-a2ae-86146375d79e 2024-09-22T00:32:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:32:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2333764.27,"bytes_per_snapshot":70012928.0,"last_snapshot_bytes":70012928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"} 2024-09-22T00:32:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:32:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:32:05 2024-09-22T00:32:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:32:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:32:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:32:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:32:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:32:05' 2024-09-22T00:32:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:32:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:32:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:32:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2333764.27,"bytes_per_snapshot":70012928.0,"last_snapshot_bytes":70012928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"} 2024-09-22T00:32:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:32:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:32:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:32:08.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:32:08.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:32:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:32:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:32:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:32:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:32:05' 2024-09-22T00:32:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-22 00:32:05' 2024-09-22T00:32:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:32:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:32:05' 2024-09-22T00:32:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:32:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:32:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:32:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK742614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1039691262-dc09-459e-a706-083041ddeb1bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65003520.0,"last_snapshot_bytes":65003520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:05image-primary4b5dc0f6c-1ed1-4c7a-a2ae-86146375d79eup+replayingreplaying, {"bytes_per_second":2333764.27,"bytes_per_snapshot":70012928.0,"last_snapshot_bytes":70012928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:05image-primary58da3684b-bb34-4654-9135-6980fd409d92up+replayingreplaying, {"bytes_per_second":2398208.0,"bytes_per_snapshot":71946240.0,"last_snapshot_bytes":71946240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965085,"remote_snapshot_timestamp":1726965085,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:05image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+replayingreplaying, {"bytes_per_second":2406400.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:05image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+replayingreplaying, {"bytes_per_second":2393429.33,"bytes_per_snapshot":71802880.0,"last_snapshot_bytes":71802880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:05image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:05image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:05' 2024-09-22T00:32:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:32:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:32:08.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:32:05' 2024-09-22T00:32:08.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:32:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:32:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:32:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:32:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:32:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-22T00:32:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:32:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:32:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-22T00:32:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-22T00:32:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:32:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-22T00:32:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:32:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:32:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary4 2024-09-22T00:32:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary4 2024-09-22T00:32:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-22T00:32:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-22T00:32:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:32:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:32:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-22T00:32:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:32:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:32:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:32:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:32:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-22T00:32:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:32:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:32:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:32:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:32:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:32:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:32:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:32:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8da3684b-bb34-4654-9135-6980fd409d92 2024-09-22T00:32:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:32:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2398208.0,"bytes_per_snapshot":71946240.0,"last_snapshot_bytes":71946240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965085,"remote_snapshot_timestamp":1726965085,"replay_state":"idle"} 2024-09-22T00:32:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:32:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:32:05 2024-09-22T00:32:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:32:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:32:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:32:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:32:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:32:10' 2024-09-22T00:32:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:32:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:32:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:32:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2398208.0,"bytes_per_snapshot":71946240.0,"last_snapshot_bytes":71946240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965085,"remote_snapshot_timestamp":1726965085,"replay_state":"idle"} 2024-09-22T00:32:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:32:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:32:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:32:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:32:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:32:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:32:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:32:11.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:32:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:32:05' 2024-09-22T00:32:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-22 00:32:05' 2024-09-22T00:32:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:32:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:32:05' 2024-09-22T00:32:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:32:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:32:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:32:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK642614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1039691262-dc09-459e-a706-083041ddeb1bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65003520.0,"last_snapshot_bytes":65003520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:10image-primary58da3684b-bb34-4654-9135-6980fd409d92up+replayingreplaying, {"bytes_per_second":2398208.0,"bytes_per_snapshot":71946240.0,"last_snapshot_bytes":71946240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965085,"remote_snapshot_timestamp":1726965085,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:10image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+replayingreplaying, {"bytes_per_second":2406400.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:10image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+replayingreplaying, {"bytes_per_second":2393429.33,"bytes_per_snapshot":71802880.0,"last_snapshot_bytes":71802880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:10image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:10image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:10' 2024-09-22T00:32:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:32:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:32:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:32:05' 2024-09-22T00:32:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:32:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:32:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:32:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:32:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:32:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-22T00:32:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:32:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:32:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-22T00:32:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-22T00:32:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:32:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-22T00:32:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:32:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:32:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary5 2024-09-22T00:32:13.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary5 2024-09-22T00:32:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-22T00:32:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-22T00:32:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:32:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:32:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-22T00:32:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:32:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:32:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:32:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:32:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-22T00:32:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:32:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:32:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:32:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:32:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:32:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:32:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:32:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 3623ca88-98f0-4728-9ca6-b784db8b4075 2024-09-22T00:32:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:32:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2406400.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"} 2024-09-22T00:32:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:32:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:32:05 2024-09-22T00:32:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:32:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:32:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:32:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:32:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:32:10' 2024-09-22T00:32:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:32:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:32:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:32:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2406400.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"} 2024-09-22T00:32:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:32:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:32:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:32:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:32:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:32:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:32:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:32:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:32:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:32:05' 2024-09-22T00:32:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-22 00:32:05' 2024-09-22T00:32:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:32:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:32:05' 2024-09-22T00:32:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:32:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:32:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:32:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK542614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1039691262-dc09-459e-a706-083041ddeb1bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65003520.0,"last_snapshot_bytes":65003520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:10image-primary63623ca88-98f0-4728-9ca6-b784db8b4075up+replayingreplaying, {"bytes_per_second":2406400.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:10image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+replayingreplaying, {"bytes_per_second":2393429.33,"bytes_per_snapshot":71802880.0,"last_snapshot_bytes":71802880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:10image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:10image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:10' 2024-09-22T00:32:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:32:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:32:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:32:05' 2024-09-22T00:32:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:32:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:32:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:32:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:32:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:32:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-22T00:32:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:32:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:32:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-22T00:32:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-22T00:32:15.063 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:32:15.060+0000 7fab5fe19640 -1 reset not still connected to 0x562403421c70 2024-09-22T00:32:15.063 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:32:15.060+0000 7fab5fe19640 -1 reset not still connected to 0x562407cc4410 2024-09-22T00:32:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:32:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-22T00:32:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:32:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:32:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary6 2024-09-22T00:32:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary6 2024-09-22T00:32:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-22T00:32:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-22T00:32:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:32:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:32:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-22T00:32:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:32:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:32:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:32:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:32:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-22T00:32:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:32:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:32:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:32:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:32:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:32:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:32:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:32:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 375b2e3e-524e-40f4-a41f-19efac72a51a 2024-09-22T00:32:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:32:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2393429.33,"bytes_per_snapshot":71802880.0,"last_snapshot_bytes":71802880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"} 2024-09-22T00:32:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:32:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:32:05 2024-09-22T00:32:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:32:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:32:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:32:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:32:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:32:15' 2024-09-22T00:32:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:32:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:32:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:32:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2393429.33,"bytes_per_snapshot":71802880.0,"last_snapshot_bytes":71802880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"} 2024-09-22T00:32:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:32:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:32:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:32:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:32:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:32:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:32:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:32:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:32:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:32:05' 2024-09-22T00:32:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-22 00:32:05' 2024-09-22T00:32:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:32:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:32:05' 2024-09-22T00:32:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:32:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:32:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:32:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK442614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1039691262-dc09-459e-a706-083041ddeb1bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65003520.0,"last_snapshot_bytes":65003520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:15image-primary7375b2e3e-524e-40f4-a41f-19efac72a51aup+replayingreplaying, {"bytes_per_second":2393429.33,"bytes_per_snapshot":71802880.0,"last_snapshot_bytes":71802880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965086,"remote_snapshot_timestamp":1726965086,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:15image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:15image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:15' 2024-09-22T00:32:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:32:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:32:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:32:05' 2024-09-22T00:32:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:32:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:32:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:32:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:32:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:32:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-22T00:32:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:32:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:32:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-22T00:32:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-22T00:32:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:32:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-22T00:32:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:32:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:32:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary7 2024-09-22T00:32:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary7 2024-09-22T00:32:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-22T00:32:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-22T00:32:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:32:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:32:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-22T00:32:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:32:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:32:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:32:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:32:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-22T00:32:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:32:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:32:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:32:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:32:20.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:32:20.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:32:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:32:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6d02ae97-ffa3-463b-bb5f-e1ee89d8d122 2024-09-22T00:32:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:32:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"} 2024-09-22T00:32:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:32:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:32:05 2024-09-22T00:32:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:32:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:32:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:32:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:32:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:32:20' 2024-09-22T00:32:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:32:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:32:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:32:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"} 2024-09-22T00:32:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:32:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:32:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:32:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:32:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:32:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:32:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:32:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:32:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:32:05' 2024-09-22T00:32:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-22 00:32:05' 2024-09-22T00:32:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:32:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:32:05' 2024-09-22T00:32:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:32:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:32:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:32:20.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1039691262-dc09-459e-a706-083041ddeb1bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65003520.0,"last_snapshot_bytes":65003520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:20image-primary86d02ae97-ffa3-463b-bb5f-e1ee89d8d122up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:20image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:20' 2024-09-22T00:32:20.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:32:20.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:32:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:32:05' 2024-09-22T00:32:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:32:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:32:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:32:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:32:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:32:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-22T00:32:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:32:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:32:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-22T00:32:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-22T00:32:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:32:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-22T00:32:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:32:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:32:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary8 2024-09-22T00:32:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary8 2024-09-22T00:32:22.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:22.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-22T00:32:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-22T00:32:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:32:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:32:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-22T00:32:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:32:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:32:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:32:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:32:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-22T00:32:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:32:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:32:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:32:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:32:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:32:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:32:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:32:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e11fe15e-b1f5-4761-b5de-fa0922f20c54 2024-09-22T00:32:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:32:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"} 2024-09-22T00:32:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:32:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:32:05 2024-09-22T00:32:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:32:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:32:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:32:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:32:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:32:20' 2024-09-22T00:32:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:32:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:32:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:32:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"} 2024-09-22T00:32:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:32:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:32:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:32:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:32:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:32:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:32:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:32:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:32:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:32:05' 2024-09-22T00:32:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-22 00:32:05' 2024-09-22T00:32:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:32:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:32:05' 2024-09-22T00:32:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:32:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:32:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:32:24.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK242614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1039691262-dc09-459e-a706-083041ddeb1bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65003520.0,"last_snapshot_bytes":65003520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:20image-primary9e11fe15e-b1f5-4761-b5de-fa0922f20c54up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:20' 2024-09-22T00:32:24.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:32:24.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:32:24.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:32:05' 2024-09-22T00:32:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:32:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:32:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:32:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:32:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:32:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-22T00:32:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:32:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:32:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-22T00:32:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-22T00:32:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:32:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-22T00:32:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:32:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:32:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary9 2024-09-22T00:32:25.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:25.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary9 2024-09-22T00:32:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-22T00:32:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-22T00:32:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:32:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:32:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-22T00:32:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:32:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:32:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:32:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:32:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-22T00:32:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:32:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:32:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:32:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:32:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:32:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:32:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:32:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 39691262-dc09-459e-a706-083041ddeb1b 2024-09-22T00:32:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:32:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65003520.0,"last_snapshot_bytes":65003520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"} 2024-09-22T00:32:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:32:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:32:05 2024-09-22T00:32:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:32:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:32:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:32:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:32:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:32:25' 2024-09-22T00:32:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:32:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:32:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:32:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65003520.0,"last_snapshot_bytes":65003520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"} 2024-09-22T00:32:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:32:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:32:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:32:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:32:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:32:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:32:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:32:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:32:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:32:05' 2024-09-22T00:32:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-22 00:32:05' 2024-09-22T00:32:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:32:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:32:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:32:05' 2024-09-22T00:32:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:32:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:32:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:32:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK142614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1039691262-dc09-459e-a706-083041ddeb1bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65003520.0,"last_snapshot_bytes":65003520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965079,"remote_snapshot_timestamp":1726965079,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:32:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:32:25' 2024-09-22T00:32:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:32:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:32:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:32:05' 2024-09-22T00:32:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:32:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:32:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:32:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:32:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:32:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-22T00:32:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:32:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:32:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-22T00:32:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-22T00:32:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:32:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-22T00:32:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:32:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:32:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:32:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary10 2024-09-22T00:32:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary10 2024-09-22T00:32:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..10} 2024-09-22T00:32:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS=() 2024-09-22T00:32:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS=() 2024-09-22T00:32:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS=() 2024-09-22T00:32:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS=() 2024-09-22T00:32:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ RET=0 2024-09-22T00:32:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-22T00:32:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary1 2024-09-22T00:32:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-22T00:32:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:32:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:32:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-22T00:32:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:32:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:32:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:32:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:32:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-22T00:32:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:32:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:32:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-22T00:32:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:32:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-22T00:32:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:32:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-22T00:32:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary1': 2024-09-22T00:32:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:32:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:32:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:32:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2b7ffc4460b2 2024-09-22T00:32:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2b7ffc4460b2 2024-09-22T00:32:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:32:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:32:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:32:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:32:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:32:28 2024 2024-09-22T00:32:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:32:28 2024 2024-09-22T00:32:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:32:28 2024 2024-09-22T00:32:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:32:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:32:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: a360adf9-9dc4-42ed-b481-6c2a0fe2c78a 2024-09-22T00:32:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:32:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:32:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:32:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-22T00:32:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd0 2024-09-22T00:32:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:32:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-22T00:32:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:32:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:32:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:32:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 033db967-f190-4b0d-91ba-ec4c308ba733 2024-09-22T00:32:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:32:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:32:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:32:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:32:29.852 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:32:29.844+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281520 2024-09-22T00:32:29.852 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:32:29.844+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:32:29.852 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:32:29.848+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T00:32:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:32:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:32:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary1 2024-09-22T00:32:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-22T00:32:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:32:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary1 2024-09-22T00:32:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:32:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-22T00:32:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:32:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary1 2024-09-22T00:32:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:32:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-22T00:32:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary2 2024-09-22T00:32:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:32:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-22T00:32:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary1 2024-09-22T00:32:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:32:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-22T00:32:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:32:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:32:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:32:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-22T00:32:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:32:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:32:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:32:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:32:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-22T00:32:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:32:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:32:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-22T00:32:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:32:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-22T00:32:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:32:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1986 2024-09-22T00:32:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-22T00:32:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary2': 2024-09-22T00:32:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:32:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:32:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:32:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2b8ea8da77d4 2024-09-22T00:32:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2b8ea8da77d4 2024-09-22T00:32:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:32:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:32:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:32:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:32:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:32:30 2024 2024-09-22T00:32:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:32:30 2024 2024-09-22T00:32:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:32:30 2024 2024-09-22T00:32:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:32:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:32:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 953d8d1a-423b-404a-bfe8-3deffbfa74d4 2024-09-22T00:32:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:32:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:32:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:32:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-22T00:32:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd1 2024-09-22T00:32:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:32:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-22T00:32:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:32:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:32:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:32:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: a1328a7e-5603-4f15-bdbe-4a1d46ac0c1e 2024-09-22T00:32:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:32:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:32:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:32:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:32:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:32:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:32:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary2 2024-09-22T00:32:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-22T00:32:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:32:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-22T00:32:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:32:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary2 2024-09-22T00:32:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:32:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-22T00:32:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:32:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:32:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:32:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:32:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary2 2024-09-22T00:32:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:32:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-22T00:32:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary3 2024-09-22T00:32:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-22T00:32:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:32:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary2 2024-09-22T00:32:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:32:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-22T00:32:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-22T00:32:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:32:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:32:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:32:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:32:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-22T00:32:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:32:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:32:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-22T00:32:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:32:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-22T00:32:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:32:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-22T00:32:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:32:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1988 2024-09-22T00:32:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-22T00:32:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary3': 2024-09-22T00:32:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:32:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:32:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:32:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2b9e69c4be6d 2024-09-22T00:32:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2b9e69c4be6d 2024-09-22T00:32:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:32:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:32:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:32:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:32:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:32:31 2024 2024-09-22T00:32:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:32:31 2024 2024-09-22T00:32:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:32:31 2024 2024-09-22T00:32:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:32:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:32:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: db5a3f2b-5b3d-4185-a630-858f70204abb 2024-09-22T00:32:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:32:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:32:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:32:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-22T00:32:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd2 2024-09-22T00:32:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:32:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-22T00:32:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:32:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:32:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:32:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 114dba15-9ab5-4b5e-8f3f-443dfab8f1e8 2024-09-22T00:32:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:32:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:32:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:32:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:32:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:32:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:32:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary3 2024-09-22T00:32:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-22T00:32:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:32:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary3 2024-09-22T00:32:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:32:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-22T00:32:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:32:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:32:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-22T00:32:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary4 2024-09-22T00:32:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:32:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-22T00:32:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:32:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:32:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:32:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:32:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-22T00:32:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary3 2024-09-22T00:32:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:32:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:32:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:32:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:32:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-22T00:32:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary3 2024-09-22T00:32:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-22T00:32:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:32:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-22T00:32:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:32:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:32:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-22T00:32:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:32:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-22T00:32:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:32:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1990 2024-09-22T00:32:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-22T00:32:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary4': 2024-09-22T00:32:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:32:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:32:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:32:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2bad67997c13 2024-09-22T00:32:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2bad67997c13 2024-09-22T00:32:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:32:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:32:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:32:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:32:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:32:32 2024 2024-09-22T00:32:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:32:32 2024 2024-09-22T00:32:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:32:32 2024 2024-09-22T00:32:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:32:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:32:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 005a21ea-613d-4f8c-8c61-abb670ab6b3f 2024-09-22T00:32:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:32:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:32:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:32:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-22T00:32:32.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd3 2024-09-22T00:32:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:32:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-22T00:32:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:32:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:32:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:32:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 3d2d12fa-00f9-4713-b21a-138fa2d42b30 2024-09-22T00:32:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:32:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:32:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:32:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:32:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:32:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:32:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary4 2024-09-22T00:32:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-22T00:32:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:32:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary4 2024-09-22T00:32:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:32:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-22T00:32:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:32:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:32:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:32:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:32:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:32:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-22T00:32:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary5 2024-09-22T00:32:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-22T00:32:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:32:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary4 2024-09-22T00:32:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:32:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-22T00:32:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:32:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:32:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:32:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:32:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-22T00:32:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary4 2024-09-22T00:32:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-22T00:32:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:32:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:32:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-22T00:32:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:32:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-22T00:32:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:32:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-22T00:32:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1993 2024-09-22T00:32:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:32:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:32:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:32:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:32:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-22T00:32:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary5': 2024-09-22T00:32:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:32:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:32:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:32:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2bb9a90a9250 2024-09-22T00:32:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2bb9a90a9250 2024-09-22T00:32:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:32:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:32:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:32:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:32:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:32:32 2024 2024-09-22T00:32:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:32:32 2024 2024-09-22T00:32:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:32:32 2024 2024-09-22T00:32:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:32:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:32:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 778f8891-97b2-4db6-a75e-3094d57bdd92 2024-09-22T00:32:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:32:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:32:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:32:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-22T00:32:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd4 2024-09-22T00:32:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:32:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-22T00:32:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:32:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:32:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:32:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: a8f3b354-141e-4248-b62e-b69050d727fe 2024-09-22T00:32:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:32:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:32:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:32:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:32:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:32:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:32:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary5 2024-09-22T00:32:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-22T00:32:33.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:32:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary5 2024-09-22T00:32:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:32:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-22T00:32:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:32:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:32:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:32:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:32:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:32:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary5 2024-09-22T00:32:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-22T00:32:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary6 2024-09-22T00:32:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-22T00:32:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:32:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary5 2024-09-22T00:32:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:32:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-22T00:32:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-22T00:32:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:32:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:32:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:32:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:32:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-22T00:32:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:32:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:32:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-22T00:32:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:32:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-22T00:32:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:32:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-22T00:32:34.213 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:32:34.212+0000 7fab5fe19640 -1 reset not still connected to 0x5623f97bd860 2024-09-22T00:32:34.218 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:32:34.212+0000 7fab5fe19640 -1 reset not still connected to 0x5623f97bdba0 2024-09-22T00:32:34.218 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:32:34.212+0000 7fab5fe19640 -1 reset not still connected to 0x5623fadb52b0 2024-09-22T00:32:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1996 2024-09-22T00:32:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:32:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-22T00:32:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:32:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:32:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:32:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary6': 2024-09-22T00:32:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:32:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:32:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:32:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2bccfd5f1023 2024-09-22T00:32:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2bccfd5f1023 2024-09-22T00:32:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:32:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:32:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:32:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:32:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:32:33 2024 2024-09-22T00:32:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:32:33 2024 2024-09-22T00:32:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:32:33 2024 2024-09-22T00:32:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:32:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:32:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 0d381d63-1eb0-4a28-89be-01bd007e4f27 2024-09-22T00:32:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:32:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:32:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:32:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-22T00:32:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd5 2024-09-22T00:32:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:32:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-22T00:32:34.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:32:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:32:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:32:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 54cc4f44-e1e6-45e0-b546-b3ce38ca8ccf 2024-09-22T00:32:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:32:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:32:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:32:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:32:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:32:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:32:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary6 2024-09-22T00:32:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-22T00:32:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:32:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary6 2024-09-22T00:32:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:32:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-22T00:32:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:32:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:32:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-22T00:32:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:32:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary7 2024-09-22T00:32:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary6 2024-09-22T00:32:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-22T00:32:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:32:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:32:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:32:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:32:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary6 2024-09-22T00:32:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-22T00:32:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-22T00:32:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:32:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:32:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:32:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:32:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-22T00:32:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:32:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:32:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-22T00:32:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:32:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-22T00:32:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:32:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-22T00:32:35.028 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:32:35.028+0000 7f54f2d95640 -1 reset not still connected to 0x5617f324f450 2024-09-22T00:32:35.029 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:32:35.028+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281450 2024-09-22T00:32:35.030 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:32:35.028+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281520 2024-09-22T00:32:35.030 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:32:35.028+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:32:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1994 2024-09-22T00:32:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:32:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:32:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:32:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:32:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-22T00:32:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1998 2024-09-22T00:32:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary7': 2024-09-22T00:32:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:32:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:32:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:32:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2bdde541236 2024-09-22T00:32:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2bdde541236 2024-09-22T00:32:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:32:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:32:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:32:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:32:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:32:34 2024 2024-09-22T00:32:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:32:34 2024 2024-09-22T00:32:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:32:34 2024 2024-09-22T00:32:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:32:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:32:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: f10a60d7-c2e8-4819-ac0b-9abd9de85975 2024-09-22T00:32:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:32:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:32:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:32:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-22T00:32:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd6 2024-09-22T00:32:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:32:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-22T00:32:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:32:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:32:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:32:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: ee3ce47f-60bd-4aed-8f33-328741cf6a37 2024-09-22T00:32:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:32:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:32:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:32:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:32:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:32:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:32:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary7 2024-09-22T00:32:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-22T00:32:35.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:32:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary7 2024-09-22T00:32:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:32:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:32:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-22T00:32:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-22T00:32:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary8 2024-09-22T00:32:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-22T00:32:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:32:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:32:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary7 2024-09-22T00:32:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-22T00:32:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:32:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:32:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:32:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:32:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:32:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:32:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-22T00:32:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:32:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:32:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary7 2024-09-22T00:32:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-22T00:32:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:32:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:32:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-22T00:32:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:32:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-22T00:32:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:32:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-22T00:32:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 1997 2024-09-22T00:32:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:32:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:32:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:32:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:32:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2001 2024-09-22T00:32:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-22T00:32:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary8': 2024-09-22T00:32:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:32:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:32:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:32:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2bef207e087c 2024-09-22T00:32:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2bef207e087c 2024-09-22T00:32:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:32:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:32:36 2024 2024-09-22T00:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:32:36 2024 2024-09-22T00:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:32:36 2024 2024-09-22T00:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 44563834-6c39-4532-8b94-956e536c455f 2024-09-22T00:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:32:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:32:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:32:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-22T00:32:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd7 2024-09-22T00:32:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:32:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-22T00:32:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:32:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:32:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:32:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 37d78258-f1b2-4d0f-b047-8fef2848394d 2024-09-22T00:32:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:32:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:32:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:32:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:32:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:32:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:32:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary8 2024-09-22T00:32:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-22T00:32:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:32:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary8 2024-09-22T00:32:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:32:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-22T00:32:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:32:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:32:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:32:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:32:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:32:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary8 2024-09-22T00:32:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-22T00:32:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary9 2024-09-22T00:32:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-22T00:32:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary8 2024-09-22T00:32:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-22T00:32:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:32:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:32:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-22T00:32:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:32:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:32:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:32:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:32:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-22T00:32:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:32:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:32:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-22T00:32:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:32:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-22T00:32:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:32:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-22T00:32:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2000 2024-09-22T00:32:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:32:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:32:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:32:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:32:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2004 2024-09-22T00:32:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-22T00:32:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary9': 2024-09-22T00:32:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:32:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:32:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:32:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2c011322340a 2024-09-22T00:32:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2c011322340a 2024-09-22T00:32:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:32:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:32:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:32:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:32:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:32:36 2024 2024-09-22T00:32:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:32:36 2024 2024-09-22T00:32:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:32:36 2024 2024-09-22T00:32:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:32:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:32:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 638cdf02-b978-469f-a12a-a6548054f4f4 2024-09-22T00:32:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:32:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:32:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:32:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-22T00:32:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd8 2024-09-22T00:32:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:32:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-22T00:32:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:32:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:32:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:32:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 1aecffb5-ff24-4807-be9e-d81a07b9c68f 2024-09-22T00:32:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:32:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:32:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:32:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:32:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:32:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:32:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:37.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary9 2024-09-22T00:32:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-22T00:32:37.989 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:32:37.988+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:32:37.994 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:32:37.992+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T00:32:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:32:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary9 2024-09-22T00:32:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:32:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-22T00:32:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:32:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:32:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-22T00:32:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary10 2024-09-22T00:32:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-22T00:32:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:32:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary9 2024-09-22T00:32:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:32:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:32:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:32:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:32:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:32:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-22T00:32:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:32:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:32:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:32:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:32:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:32:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-22T00:32:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary9 2024-09-22T00:32:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-22T00:32:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:32:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:32:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:32:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-22T00:32:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:32:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:32:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-22T00:32:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:32:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-22T00:32:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2003 2024-09-22T00:32:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:32:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:32:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:32:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:32:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2007 2024-09-22T00:32:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-22T00:32:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:32:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:32:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:32:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary10': 2024-09-22T00:32:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:32:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:32:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:32:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2c12b01a327d 2024-09-22T00:32:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2c12b01a327d 2024-09-22T00:32:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:32:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:32:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:32:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:32:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:32:38 2024 2024-09-22T00:32:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:32:38 2024 2024-09-22T00:32:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:32:38 2024 2024-09-22T00:32:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:32:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:32:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 8199a7a1-0f0f-4500-8687-d6ef1ddb91c2 2024-09-22T00:32:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:32:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:32:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:32:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-22T00:32:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd9 2024-09-22T00:32:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:32:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-22T00:32:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:32:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:32:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:32:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 794987d8-ebe4-491d-b4d9-59cc05eb9161 2024-09-22T00:32:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:32:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:32:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:32:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:32:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:32:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:32:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:32:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary10 2024-09-22T00:32:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-22T00:32:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:32:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary10 2024-09-22T00:32:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:32:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-22T00:32:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:32:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:32:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:32:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 98108 2024-09-22T00:32:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:32:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:32:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:32:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary10 2024-09-22T00:32:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary10 2024-09-22T00:32:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-22T00:32:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:32:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-22T00:32:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2006 2024-09-22T00:32:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2011 2024-09-22T00:32:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:32:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:32:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:32:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:32:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:32:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:32:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2009 2024-09-22T00:32:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2010 2024-09-22T00:32:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:32:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:32:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:32:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:32:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:32:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:32:41.365 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:32:41.361+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a36a91e410 2024-09-22T00:32:41.365 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:32:41.361+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a3630960d0 2024-09-22T00:32:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2013 2024-09-22T00:32:41.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2012 2024-09-22T00:32:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:32:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:32:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:32:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:32:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:32:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:32:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:32:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:32:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:32:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2015 2024-09-22T00:32:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2014 2024-09-22T00:32:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:32:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:32:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:32:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2017 2024-09-22T00:32:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2016 2024-09-22T00:32:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:32:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:32:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:32:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:32:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:32:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:32:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2018 2024-09-22T00:32:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2019 2024-09-22T00:32:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:32:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:32:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:32:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:32:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:32:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:32:45.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2020 2024-09-22T00:32:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2021 2024-09-22T00:32:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:32:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:32:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:32:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:32:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:32:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:32:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:45.839 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:32:45.835+0000 7fab5fe19640 -1 reset not still connected to 0x5623f97bd860 2024-09-22T00:32:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2022 2024-09-22T00:32:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2023 2024-09-22T00:32:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:32:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:32:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:32:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:46.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:32:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:32:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:32:46.964 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:32:46.963+0000 7f54f2d95640 -1 reset not still connected to 0x5617f4767860 2024-09-22T00:32:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:32:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:32:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:32:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2024 2024-09-22T00:32:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:32:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:32:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:32:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2025 2024-09-22T00:32:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2026 2024-09-22T00:32:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2027 2024-09-22T00:32:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:32:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:32:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:32:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:32:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:32:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:32:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2028 2024-09-22T00:32:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2029 2024-09-22T00:32:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:32:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:32:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:32:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:32:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:32:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:32:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2031 2024-09-22T00:32:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2030 2024-09-22T00:32:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:32:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:32:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:32:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:32:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:32:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:32:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2033 2024-09-22T00:32:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2032 2024-09-22T00:32:51.498 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:32:51.495+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281450 2024-09-22T00:32:51.498 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:32:51.495+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281520 2024-09-22T00:32:51.498 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:32:51.495+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:32:51.501 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:32:51.499+0000 7f54f2d95640 -1 reset not still connected to 0x5618000d0410 2024-09-22T00:32:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:32:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:32:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:32:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:32:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:32:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:32:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:32:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:32:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:32:52.420 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:32:52.412+0000 7f84953c5640 -1 reset not still connected to 0x5636f8ec80d0 2024-09-22T00:32:52.420 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:32:52.416+0000 7f84953c5640 -1 reset not still connected to 0x5636f9c289c0 2024-09-22T00:32:52.420 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:32:52.416+0000 7f84953c5640 -1 reset not still connected to 0x5636ff350d00 2024-09-22T00:32:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:32:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:32:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:32:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2034 2024-09-22T00:32:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2035 2024-09-22T00:32:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2037 2024-09-22T00:32:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2036 2024-09-22T00:32:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:32:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:32:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:32:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:53.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:32:53.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:53.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:53.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:32:53.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:32:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2038 2024-09-22T00:32:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2039 2024-09-22T00:32:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:32:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:32:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:32:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:32:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:32:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:32:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2040 2024-09-22T00:32:55.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2041 2024-09-22T00:32:55.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:55.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:32:55.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:55.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:55.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:32:55.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:32:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:32:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:32:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:32:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:56.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2042 2024-09-22T00:32:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2043 2024-09-22T00:32:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:32:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:32:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:32:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:32:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:32:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:32:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:32:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:32:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:32:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:32:57.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:57.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:57.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:32:57.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:32:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2045 2024-09-22T00:32:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2044 2024-09-22T00:32:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:58.530 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:32:58.527+0000 7f54f2d95640 -1 reset not still connected to 0x561804597c70 2024-09-22T00:32:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2046 2024-09-22T00:32:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:32:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:32:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:32:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:32:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:32:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:32:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2048 2024-09-22T00:32:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2049 2024-09-22T00:32:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2047 2024-09-22T00:32:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:32:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:32:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:32:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:32:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:32:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:32:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:32:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:32:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:32:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:33:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2050 2024-09-22T00:33:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2051 2024-09-22T00:33:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:33:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:33:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:33:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:00.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:00.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:33:00.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:00.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:00.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:33:00.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:33:00.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2053 2024-09-22T00:33:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2052 2024-09-22T00:33:01.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:01.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:33:01.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:01.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:01.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:33:01.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:33:01.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:01.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:33:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:33:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:33:02.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:02.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:33:02.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:02.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:02.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:33:02.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:33:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:33:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:33:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:33:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2055 2024-09-22T00:33:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2054 2024-09-22T00:33:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:03.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2056 2024-09-22T00:33:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:33:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:33:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:33:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:03.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2057 2024-09-22T00:33:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:33:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:33:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:33:03.606 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:33:03.603+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3392ea0 2024-09-22T00:33:03.609 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:33:03.607+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3393110 2024-09-22T00:33:03.610 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:33:03.607+0000 7f54f2d95640 -1 reset not still connected to 0x5617f33931e0 2024-09-22T00:33:03.610 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:33:03.607+0000 7f54f2d95640 -1 reset not still connected to 0x5617fb1e81a0 2024-09-22T00:33:03.610 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:33:03.607+0000 7f54f2d95640 -1 reset not still connected to 0x561800fa2750 2024-09-22T00:33:03.610 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:33:03.607+0000 7f54f2d95640 -1 reset not still connected to 0x56180927b6c0 2024-09-22T00:33:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2058 2024-09-22T00:33:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2059 2024-09-22T00:33:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:33:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:33:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:33:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:33:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:33:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:33:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2060 2024-09-22T00:33:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2061 2024-09-22T00:33:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:33:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:33:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:33:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:33:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:33:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:33:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2063 2024-09-22T00:33:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2062 2024-09-22T00:33:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:33:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:33:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:33:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:33:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:33:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:33:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2064 2024-09-22T00:33:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2065 2024-09-22T00:33:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:33:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:33:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:33:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:33:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:33:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:33:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2067 2024-09-22T00:33:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:33:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:33:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:33:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:33:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:33:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:33:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2070 2024-09-22T00:33:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2068 2024-09-22T00:33:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2069 2024-09-22T00:33:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:33:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:33:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:33:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:33:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:33:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:33:09.534 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:33:09.531+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7f6286270 2024-09-22T00:33:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2072 2024-09-22T00:33:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2071 2024-09-22T00:33:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:33:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:33:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:33:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:33:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:33:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:33:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:10.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2073 2024-09-22T00:33:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2074 2024-09-22T00:33:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:33:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:33:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:33:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2076 2024-09-22T00:33:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2075 2024-09-22T00:33:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:33:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:33:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:33:12.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:33:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:33:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:33:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:33:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:33:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:33:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2077 2024-09-22T00:33:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:33:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:33:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:33:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:33:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:13.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:33:13.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:33:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2078 2024-09-22T00:33:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2079 2024-09-22T00:33:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2080 2024-09-22T00:33:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:33:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:33:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:33:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:33:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:33:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:33:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2081 2024-09-22T00:33:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2082 2024-09-22T00:33:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:33:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:33:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:33:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:33:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:33:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:33:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2083 2024-09-22T00:33:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2084 2024-09-22T00:33:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:33:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:33:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:33:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2085 2024-09-22T00:33:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2086 2024-09-22T00:33:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:33:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:33:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:33:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:33:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:33:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:33:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:33:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:33:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:33:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2087 2024-09-22T00:33:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:33:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:33:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:33:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:33:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:33:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:33:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2090 2024-09-22T00:33:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2088 2024-09-22T00:33:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2089 2024-09-22T00:33:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:19.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:33:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:33:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:33:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:33:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:33:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:33:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2091 2024-09-22T00:33:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2092 2024-09-22T00:33:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:33:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:33:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:33:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:33:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:33:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:33:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2093 2024-09-22T00:33:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2094 2024-09-22T00:33:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:33:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:33:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:33:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2096 2024-09-22T00:33:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:33:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:33:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:33:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:33:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:33:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:33:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:33:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:33:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:33:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2095 2024-09-22T00:33:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:23.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:33:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:33:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:33:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:33:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:33:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:33:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2098 2024-09-22T00:33:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2099 2024-09-22T00:33:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2097 2024-09-22T00:33:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2100 2024-09-22T00:33:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:33:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:33:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:33:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:33:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:33:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:33:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2101 2024-09-22T00:33:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:33:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:33:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:33:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2102 2024-09-22T00:33:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:33:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:33:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:33:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2104 2024-09-22T00:33:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2103 2024-09-22T00:33:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:33:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:33:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:33:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:33:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:33:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:33:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:33:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:33:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:33:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:33:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:33:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:33:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2105 2024-09-22T00:33:27.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2106 2024-09-22T00:33:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2108 2024-09-22T00:33:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:33:28.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:28.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:28.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:33:28.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:33:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:33:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:33:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:33:28.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2109 2024-09-22T00:33:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2107 2024-09-22T00:33:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2110 2024-09-22T00:33:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:33:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:33:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:33:29.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:29.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:33:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:33:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:33:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2112 2024-09-22T00:33:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2111 2024-09-22T00:33:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:33:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:33:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:33:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:33:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:33:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:33:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2113 2024-09-22T00:33:31.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2114 2024-09-22T00:33:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:33:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:33:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:33:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:33:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:33:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:33:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:33:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:33:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:33:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:33:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:33:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:33:32.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2116 2024-09-22T00:33:32.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2115 2024-09-22T00:33:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2117 2024-09-22T00:33:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:33:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:33:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:33:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:33:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:33:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:33:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2118 2024-09-22T00:33:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2119 2024-09-22T00:33:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2120 2024-09-22T00:33:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:33:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:33:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:33:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:33:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:33:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:33:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2121 2024-09-22T00:33:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2122 2024-09-22T00:33:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:33:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:33:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:33:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:33:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:33:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:33:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2123 2024-09-22T00:33:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2124 2024-09-22T00:33:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:33:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:33:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:33:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:33:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:33:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:33:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:33:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:33:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:33:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:33:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:33:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:33:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2125 2024-09-22T00:33:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2126 2024-09-22T00:33:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2127 2024-09-22T00:33:38.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:33:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:33:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:33:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:33:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:33:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:33:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2130 2024-09-22T00:33:39.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2129 2024-09-22T00:33:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2128 2024-09-22T00:33:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:33:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:33:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:33:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:33:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:33:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:33:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2132 2024-09-22T00:33:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2131 2024-09-22T00:33:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:33:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:33:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:33:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:33:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:33:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:33:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2133 2024-09-22T00:33:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2134 2024-09-22T00:33:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:33:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:33:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:33:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2136 2024-09-22T00:33:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2135 2024-09-22T00:33:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:33:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:33:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:33:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:33:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:33:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:33:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:33:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:33:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:33:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2137 2024-09-22T00:33:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:33:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:33:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:33:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:33:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:33:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:33:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2139 2024-09-22T00:33:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2138 2024-09-22T00:33:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2140 2024-09-22T00:33:44.447 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:33:44.444+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a373193930 2024-09-22T00:33:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:33:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:33:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:33:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:33:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:33:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:33:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2141 2024-09-22T00:33:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2142 2024-09-22T00:33:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:33:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:33:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:33:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:33:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:33:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:33:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2143 2024-09-22T00:33:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2144 2024-09-22T00:33:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:33:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:33:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:33:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2145 2024-09-22T00:33:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2146 2024-09-22T00:33:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:33:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:33:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:33:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:33:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:33:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:33:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:33:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:33:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:33:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2147 2024-09-22T00:33:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:33:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:33:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:33:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:48.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:33:48.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:48.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:48.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:33:48.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:33:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2149 2024-09-22T00:33:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2150 2024-09-22T00:33:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2148 2024-09-22T00:33:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:33:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:33:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:33:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:33:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:33:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:33:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2152 2024-09-22T00:33:50.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2151 2024-09-22T00:33:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:33:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:33:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:33:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:33:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:33:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:33:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2153 2024-09-22T00:33:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2154 2024-09-22T00:33:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:33:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:33:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:33:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:33:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:33:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:33:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:33:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:33:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:33:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2155 2024-09-22T00:33:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2156 2024-09-22T00:33:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:33:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:33:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:33:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2157 2024-09-22T00:33:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:33:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:33:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:33:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:33:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:33:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:33:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2160 2024-09-22T00:33:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2158 2024-09-22T00:33:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2159 2024-09-22T00:33:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:33:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:33:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:33:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:33:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:33:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:33:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2162 2024-09-22T00:33:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2161 2024-09-22T00:33:55.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:55.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:33:55.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:55.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:55.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:33:55.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:33:55.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:33:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:33:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:33:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2163 2024-09-22T00:33:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2164 2024-09-22T00:33:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:33:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:33:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:33:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:33:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:33:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:33:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:33:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:33:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:33:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:33:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:33:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:33:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2165 2024-09-22T00:33:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2166 2024-09-22T00:33:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2167 2024-09-22T00:33:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:33:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:33:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:33:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:33:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:33:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:33:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:59.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2168 2024-09-22T00:33:59.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2170 2024-09-22T00:33:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2169 2024-09-22T00:33:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:33:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:33:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:33:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:33:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:33:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:33:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:33:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:33:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:33:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:34:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2172 2024-09-22T00:34:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2171 2024-09-22T00:34:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:34:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:34:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:34:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:34:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:34:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:34:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:00.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2174 2024-09-22T00:34:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2173 2024-09-22T00:34:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:34:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:34:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:34:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:34:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:34:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:34:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:34:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:34:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:34:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:34:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:34:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:34:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2176 2024-09-22T00:34:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2175 2024-09-22T00:34:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:03.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2177 2024-09-22T00:34:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:34:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:34:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:34:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:34:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:34:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:34:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2180 2024-09-22T00:34:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2178 2024-09-22T00:34:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2179 2024-09-22T00:34:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:34:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:34:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:34:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:34:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:34:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:34:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2181 2024-09-22T00:34:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2182 2024-09-22T00:34:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:34:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:34:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:34:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:34:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:34:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:34:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2183 2024-09-22T00:34:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2184 2024-09-22T00:34:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:34:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:34:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:34:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:34:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:34:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:34:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:34:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:34:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:34:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:34:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:34:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:34:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2186 2024-09-22T00:34:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2185 2024-09-22T00:34:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2187 2024-09-22T00:34:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:34:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:34:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:34:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:34:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:34:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:34:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2189 2024-09-22T00:34:09.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2188 2024-09-22T00:34:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2190 2024-09-22T00:34:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:34:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:34:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:34:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:34:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:34:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:34:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2192 2024-09-22T00:34:10.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2191 2024-09-22T00:34:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:34:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:34:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:34:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:34:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:34:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:34:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2194 2024-09-22T00:34:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2193 2024-09-22T00:34:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:34:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:34:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:34:11.614 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:34:11.610+0000 7fab5fe19640 -1 reset not still connected to 0x562411da55f0 2024-09-22T00:34:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:34:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:34:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:34:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:34:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:34:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:34:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:34:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:34:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:34:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2195 2024-09-22T00:34:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2196 2024-09-22T00:34:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:12.612 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:34:12.608+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a36a91e410 2024-09-22T00:34:12.612 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:34:12.608+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a373193930 2024-09-22T00:34:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2197 2024-09-22T00:34:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:34:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:34:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:34:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:34:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:34:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:34:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2199 2024-09-22T00:34:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2200 2024-09-22T00:34:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2198 2024-09-22T00:34:14.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:34:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:34:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:34:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:34:14.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:14.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:14.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:34:14.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:34:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2202 2024-09-22T00:34:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2201 2024-09-22T00:34:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:34:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:34:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:34:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:34:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:34:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:34:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2203 2024-09-22T00:34:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2204 2024-09-22T00:34:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:34:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:34:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:34:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2205 2024-09-22T00:34:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2206 2024-09-22T00:34:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:34:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:34:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:34:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:34:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:34:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:34:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:34:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:34:17.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:34:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2207 2024-09-22T00:34:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:34:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:34:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:34:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:34:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:34:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:34:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2210 2024-09-22T00:34:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2208 2024-09-22T00:34:19.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2209 2024-09-22T00:34:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:34:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:34:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:34:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:34:19.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:19.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:19.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:34:19.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:34:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:19.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2212 2024-09-22T00:34:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2211 2024-09-22T00:34:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:34:20.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:20.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:20.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:34:20.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:34:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:34:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:34:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:34:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2213 2024-09-22T00:34:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2214 2024-09-22T00:34:21.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:21.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:34:21.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:21.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:21.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:34:21.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:34:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2216 2024-09-22T00:34:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2215 2024-09-22T00:34:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:34:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:34:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:34:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:34:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:34:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:34:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:34:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:34:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:34:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:23.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2217 2024-09-22T00:34:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:34:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:34:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:34:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:34:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:34:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:34:23.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2220 2024-09-22T00:34:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2219 2024-09-22T00:34:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2218 2024-09-22T00:34:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:34:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:34:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:34:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:34:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:34:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:34:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2221 2024-09-22T00:34:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2222 2024-09-22T00:34:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:34:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:34:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:34:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:34:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:34:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:34:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2224 2024-09-22T00:34:26.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2223 2024-09-22T00:34:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:34:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:34:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:34:27.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:34:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:34:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:34:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2225 2024-09-22T00:34:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2226 2024-09-22T00:34:27.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:34:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:34:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:34:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:34:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:34:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:34:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:27.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:28.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2227 2024-09-22T00:34:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:34:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:34:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:34:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:34:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:34:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:34:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2230 2024-09-22T00:34:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2229 2024-09-22T00:34:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2228 2024-09-22T00:34:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:34:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:34:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:34:29.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:29.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:34:29.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:29.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:34:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:34:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2231 2024-09-22T00:34:30.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2232 2024-09-22T00:34:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:34:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:34:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:34:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:34:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:34:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:34:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2233 2024-09-22T00:34:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2234 2024-09-22T00:34:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:34:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:34:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:34:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:34:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:34:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:34:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:34:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:34:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:34:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:34:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:34:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:34:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2236 2024-09-22T00:34:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2235 2024-09-22T00:34:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2237 2024-09-22T00:34:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:34:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:34:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:34:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:34:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:34:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:34:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2240 2024-09-22T00:34:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2239 2024-09-22T00:34:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2238 2024-09-22T00:34:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:34:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:34:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:34:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:34:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:34:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:34:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2242 2024-09-22T00:34:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:34:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:34:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:34:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:34:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:34:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:34:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2241 2024-09-22T00:34:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2243 2024-09-22T00:34:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2244 2024-09-22T00:34:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:34:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:34:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:34:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:34:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:34:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:34:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:34:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:34:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:34:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:34:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:34:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:34:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2245 2024-09-22T00:34:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2246 2024-09-22T00:34:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:37.625 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:34:37.618+0000 7fab5fe19640 -1 reset not still connected to 0x562420f39d40 2024-09-22T00:34:37.626 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:34:37.618+0000 7fab5fe19640 -1 reset not still connected to 0x562432537450 2024-09-22T00:34:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2247 2024-09-22T00:34:38.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:38.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:34:38.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:38.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:38.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:34:38.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:34:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:34:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:34:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:34:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2249 2024-09-22T00:34:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2248 2024-09-22T00:34:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2250 2024-09-22T00:34:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:34:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:34:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:34:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:34:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:34:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:34:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2252 2024-09-22T00:34:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2251 2024-09-22T00:34:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:34:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:34:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:34:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:34:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:34:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:34:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2254 2024-09-22T00:34:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2253 2024-09-22T00:34:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:34:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:34:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:34:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:34:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:34:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:34:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:34:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:34:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:34:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:34:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:34:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:34:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2256 2024-09-22T00:34:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2255 2024-09-22T00:34:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2257 2024-09-22T00:34:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:34:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:34:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:34:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:34:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:34:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:34:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2260 2024-09-22T00:34:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2259 2024-09-22T00:34:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2258 2024-09-22T00:34:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:34:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:34:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:34:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:34:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:34:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:34:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2262 2024-09-22T00:34:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2261 2024-09-22T00:34:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:34:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:34:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:34:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:34:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:34:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:34:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2263 2024-09-22T00:34:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2264 2024-09-22T00:34:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:34:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:34:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:34:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:34:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:34:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:34:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:34:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:34:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:34:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:34:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:34:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:34:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2265 2024-09-22T00:34:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2266 2024-09-22T00:34:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2267 2024-09-22T00:34:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:34:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:34:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:34:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:34:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:34:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:34:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:48.735 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:34:48.729+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7f62864e0 2024-09-22T00:34:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2270 2024-09-22T00:34:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2268 2024-09-22T00:34:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2269 2024-09-22T00:34:49.471 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:34:49.465+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7cb140340 2024-09-22T00:34:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:49.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:34:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:34:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:34:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:34:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:34:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:34:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2271 2024-09-22T00:34:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2272 2024-09-22T00:34:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:34:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:34:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:34:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:34:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:34:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:34:51.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2273 2024-09-22T00:34:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2274 2024-09-22T00:34:51.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:34:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:34:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:34:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2276 2024-09-22T00:34:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2275 2024-09-22T00:34:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:34:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:34:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:34:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:34:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:34:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:34:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:34:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:52.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:52.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:34:52.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:34:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2277 2024-09-22T00:34:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:34:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:34:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:34:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:34:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:34:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:34:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2278 2024-09-22T00:34:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2279 2024-09-22T00:34:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2280 2024-09-22T00:34:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:34:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:34:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:34:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:34:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:34:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:34:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2281 2024-09-22T00:34:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2282 2024-09-22T00:34:55.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:34:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:34:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:34:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:34:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:34:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:34:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:56.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2283 2024-09-22T00:34:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2284 2024-09-22T00:34:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:34:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:34:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:34:56.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2286 2024-09-22T00:34:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2285 2024-09-22T00:34:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:34:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:34:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:34:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:34:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:34:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:34:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:34:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:34:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:34:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2287 2024-09-22T00:34:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:34:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:34:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:34:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:34:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 98235 2024-09-22T00:34:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2290 2024-09-22T00:34:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2288 2024-09-22T00:34:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2289 2024-09-22T00:34:59.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:34:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:34:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:34:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:34:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:34:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:34:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:34:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:34:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 98362 2024-09-22T00:35:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2291 2024-09-22T00:35:00.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:35:00.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 98489 2024-09-22T00:35:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:35:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:35:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:35:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:35:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:35:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:35:00.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:35:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2292 2024-09-22T00:35:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:35:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 98636 2024-09-22T00:35:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:35:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:35:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:35:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:35:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:35:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:35:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:35:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2293 2024-09-22T00:35:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:35:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:35:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:35:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:35:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:35:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:35:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:35:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 98783 2024-09-22T00:35:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:35:03.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:35:03.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 98931 2024-09-22T00:35:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2294 2024-09-22T00:35:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2295 2024-09-22T00:35:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:35:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:35:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:35:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 99079 2024-09-22T00:35:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:35:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 99227 2024-09-22T00:35:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:35:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 99395 2024-09-22T00:35:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:35:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:35:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 98109 2024-09-22T00:37:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:37:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:37:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:37:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 98236 2024-09-22T00:37:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:37:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:37:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:37:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 98363 2024-09-22T00:37:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:37:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:37:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:37:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 98490 2024-09-22T00:37:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:37:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:37:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:37:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 98637 2024-09-22T00:37:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:37:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:37:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:37:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 98784 2024-09-22T00:37:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:37:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:37:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:37:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 98932 2024-09-22T00:37:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:37:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:37:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:37:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 99080 2024-09-22T00:37:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:37:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:37:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:37:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 99228 2024-09-22T00:37:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:37:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:37:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:37:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 99396 2024-09-22T00:37:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:37:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:37:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:37:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:37:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:37:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:37:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 1 2024-09-22T00:37:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:37:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd0 2024-09-22T00:37:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary1 2024-09-22T00:37:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary1 2024-09-22T00:37:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:37:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary1 2024-09-22T00:37:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:37:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 2 2024-09-22T00:37:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd1 2024-09-22T00:37:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary2 2024-09-22T00:37:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary2 2024-09-22T00:37:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:37:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary2 2024-09-22T00:37:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:37:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:37:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 3 2024-09-22T00:37:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 4 2024-09-22T00:37:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:37:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd2 2024-09-22T00:37:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary3 2024-09-22T00:37:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:37:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary3 2024-09-22T00:37:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:37:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary3 2024-09-22T00:37:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd3 2024-09-22T00:37:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary4 2024-09-22T00:37:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary4 2024-09-22T00:37:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:37:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary4 2024-09-22T00:37:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:37:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:37:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 5 2024-09-22T00:37:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:37:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd4 2024-09-22T00:37:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary5 2024-09-22T00:37:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:37:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary5 2024-09-22T00:37:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:37:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary5 2024-09-22T00:37:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 6 2024-09-22T00:37:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd5 2024-09-22T00:37:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:37:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:37:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 7 2024-09-22T00:37:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary6 2024-09-22T00:37:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary6 2024-09-22T00:37:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd6 2024-09-22T00:37:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary7 2024-09-22T00:37:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:37:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary7 2024-09-22T00:37:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:37:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary7 2024-09-22T00:37:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:37:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:37:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary6 2024-09-22T00:37:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:37:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:37:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:37:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:37:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 105546 2024-09-22T00:37:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 10 2024-09-22T00:37:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd9 2024-09-22T00:37:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary10 2024-09-22T00:37:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary10 2024-09-22T00:37:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:37:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary10 2024-09-22T00:37:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 9 2024-09-22T00:37:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd8 2024-09-22T00:37:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary9 2024-09-22T00:37:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary9 2024-09-22T00:37:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:37:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary9 2024-09-22T00:37:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 8 2024-09-22T00:37:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd7 2024-09-22T00:37:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary8 2024-09-22T00:37:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary8 2024-09-22T00:37:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:37:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary8 2024-09-22T00:37:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-22T00:37:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:37:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-22T00:37:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:37:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:37:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-22T00:37:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-22T00:37:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:37:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:37:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-22T00:37:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-22T00:37:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:37:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-22T00:37:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:37:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-22T00:37:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:37:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-22T00:37:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:37:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:37:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-22T00:37:46.438 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:37:46.435+0000 7f54f2d95640 -1 reset not still connected to 0x5617f4767860 2024-09-22T00:37:46.442 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:37:46.439+0000 7f54f2d95640 -1 reset not still connected to 0x5617f76b7790 2024-09-22T00:37:46.442 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:37:46.439+0000 7f54f2d95640 -1 reset not still connected to 0x5618050d5520 2024-09-22T00:37:55.505 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:37:55.502+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:37:55.511 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:37:55.506+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T00:38:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=2fe4f7c16f9855dd5560b4034aa749f6 2024-09-22T00:38:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-22T00:38:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-22T00:38:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:38:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-22T00:38:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=4848433e769e49cee289dfe98e5f5f4f 2024-09-22T00:38:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-22T00:38:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-22T00:38:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:38:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-22T00:38:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=ca9eb77a1d063da78458cb6fa0751a2e 2024-09-22T00:38:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-22T00:38:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-22T00:38:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:38:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-22T00:38:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=999a1816f1f8e32b732aa4ca4265e99a 2024-09-22T00:38:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-22T00:38:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=157fa3a6efab088615f2d07737a3ca24 2024-09-22T00:38:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-22T00:38:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-22T00:38:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:38:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-22T00:38:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-22T00:38:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:38:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-22T00:38:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=4e1e03894d556e3cc6aefe417e7814b9 2024-09-22T00:38:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-22T00:38:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=f648ca9cbec83ce919909ca5a99f71f3 2024-09-22T00:38:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-22T00:38:12.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-22T00:38:12.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:12.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:12.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:38:12.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-22T00:38:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=26e14632384bef682f27de3b068a0773 2024-09-22T00:38:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-22T00:38:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=e663780a6567413dee840fc1398da40f 2024-09-22T00:38:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-22T00:38:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-22T00:38:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:38:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-22T00:38:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:38:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-22T00:38:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:38:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=4e2a816c7e4c201c78b10853253c4a7f 2024-09-22T00:38:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-22T00:38:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-22T00:38:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:38:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-22T00:38:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-22T00:38:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:38:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-22T00:38:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-22T00:38:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:38:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-22T00:38:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:38:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:38:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-22T00:38:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:38:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:38:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:38:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:38:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-22T00:38:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:38:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-22T00:38:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:38:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-22T00:38:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:38:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-22T00:38:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:38:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:38:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f10a60d7-c2e8-4819-ac0b-9abd9de85975 2024-09-22T00:38:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:38:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:38:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:10 2024-09-22T00:38:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:38:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:38:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:13' 2024-09-22T00:38:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:38:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:38:14.785 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:38:14.781+0000 7fab1f496640 -1 reset not still connected to 0x55fa64755380 2024-09-22T00:38:14.785 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:38:14.781+0000 7fab1f496640 -1 reset not still connected to 0x55fa743b2340 2024-09-22T00:38:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:38:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:38:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:38:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:38:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-22T00:38:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:38:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:38:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-22T00:38:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:38:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-22T00:38:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:38:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-22T00:38:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:38:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:38:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-22T00:38:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:38:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:38:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-22T00:38:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:38:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-22T00:38:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:38:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:38:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-22T00:38:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:38:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:38:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-22T00:38:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:38:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:38:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 638cdf02-b978-469f-a12a-a6548054f4f4 2024-09-22T00:38:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:38:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:38:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:10 2024-09-22T00:38:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:38:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:38:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:14' 2024-09-22T00:38:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:38:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:38:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:38:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8199a7a1-0f0f-4500-8687-d6ef1ddb91c2 2024-09-22T00:38:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:38:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:38:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:10 2024-09-22T00:38:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:38:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:38:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:14' 2024-09-22T00:38:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:38:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 0d381d63-1eb0-4a28-89be-01bd007e4f27 2024-09-22T00:38:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:38:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:38:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:10 2024-09-22T00:38:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:38:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:38:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:14' 2024-09-22T00:38:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:38:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:38:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:38:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:38:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:38:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 953d8d1a-423b-404a-bfe8-3deffbfa74d4 2024-09-22T00:38:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:38:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:38:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:10 2024-09-22T00:38:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:38:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:38:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:14' 2024-09-22T00:38:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:38:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:38:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-22T00:38:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:38:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:38:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-22T00:38:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:38:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:38:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-22T00:38:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:38:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:38:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-22T00:38:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:38:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:38:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-22T00:38:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:38:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:38:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:38:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: a360adf9-9dc4-42ed-b481-6c2a0fe2c78a 2024-09-22T00:38:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:38:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:38:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:38:16.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:16.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:16.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:16.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:16.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:16.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-22T00:38:16.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:16.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:16.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:38:16.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:38:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:38:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:38:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 005a21ea-613d-4f8c-8c61-abb670ab6b3f 2024-09-22T00:38:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:38:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:38:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:38:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:38:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:38:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 778f8891-97b2-4db6-a75e-3094d57bdd92 2024-09-22T00:38:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:38:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:38:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:38:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:38:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:38:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: db5a3f2b-5b3d-4185-a630-858f70204abb 2024-09-22T00:38:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:38:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:38:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:38:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:16.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:16.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:16.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:16.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:16.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:38:16.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:16.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:16.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:16.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:16.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:38:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f10a60d7-c2e8-4819-ac0b-9abd9de85975 2024-09-22T00:38:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:38:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:38:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:38:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:38:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:38:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:38:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 44563834-6c39-4532-8b94-956e536c455f 2024-09-22T00:38:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:38:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:38:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:16.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:16.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:38:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:38:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:38:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:38:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:38:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:38:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:38:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:38:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:38:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:38:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-22T00:38:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:38:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-22T00:38:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:38:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-22T00:38:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:38:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-22T00:38:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:38:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-22T00:38:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:38:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:38:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:38:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-22T00:38:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:38:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-22T00:38:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:38:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:38:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-22T00:38:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-22T00:38:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:38:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:38:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:38:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:38:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-22T00:38:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:38:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:38:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:38:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 638cdf02-b978-469f-a12a-a6548054f4f4 2024-09-22T00:38:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:38:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:38:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:17.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:17.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:17.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:38:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:17.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:38:17.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8199a7a1-0f0f-4500-8687-d6ef1ddb91c2 2024-09-22T00:38:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:38:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:38:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 0d381d63-1eb0-4a28-89be-01bd007e4f27 2024-09-22T00:38:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:38:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:38:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:38:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:17.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:17.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:17.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:38:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:38:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:38:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 953d8d1a-423b-404a-bfe8-3deffbfa74d4 2024-09-22T00:38:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:38:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:38:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:38:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:38:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:38:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-22T00:38:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:38:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-22T00:38:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-22T00:38:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:38:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:38:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-22T00:38:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:38:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-22T00:38:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-22T00:38:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:38:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-22T00:38:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-22T00:38:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-22T00:38:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-22T00:38:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-22T00:38:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:38:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:38:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:38:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-22T00:38:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:38:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:38:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:38:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-22T00:38:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:38:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-22T00:38:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:38:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-22T00:38:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:38:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 005a21ea-613d-4f8c-8c61-abb670ab6b3f 2024-09-22T00:38:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:38:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:38:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:38:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:38:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: db5a3f2b-5b3d-4185-a630-858f70204abb 2024-09-22T00:38:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:38:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:38:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:38:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:38:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f10a60d7-c2e8-4819-ac0b-9abd9de85975 2024-09-22T00:38:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:38:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:38:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:38:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:38:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:38:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: a360adf9-9dc4-42ed-b481-6c2a0fe2c78a 2024-09-22T00:38:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:38:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:38:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:38:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 778f8891-97b2-4db6-a75e-3094d57bdd92 2024-09-22T00:38:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:38:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:38:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:38:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:38:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:38:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:38:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:38:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:38:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 44563834-6c39-4532-8b94-956e536c455f 2024-09-22T00:38:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:38:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:38:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:38:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:17.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:38:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:38:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:38:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:38:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-22T00:38:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:38:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:38:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:38:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-22T00:38:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:38:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:38:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-22T00:38:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:38:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:38:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:38:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-22T00:38:17.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:38:17.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:38:17.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-22T00:38:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:38:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:38:17.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:17.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:17.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-22T00:38:17.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:38:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:38:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-22T00:38:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:38:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:38:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:38:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-22T00:38:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:38:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:38:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-22T00:38:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:38:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-22T00:38:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:38:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:38:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-22T00:38:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:38:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-22T00:38:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:38:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:38:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-22T00:38:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-22T00:38:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:38:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-22T00:38:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:38:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:38:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:38:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:38:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:38:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:38:18.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:38:18.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:38:18.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-22T00:38:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-22T00:38:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:38:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:38:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:38:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-22T00:38:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-22T00:38:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:38:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:38:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-22T00:38:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:38:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:38:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:38:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:38:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-22T00:38:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-22T00:38:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:38:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:38:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-22T00:38:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-22T00:38:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:38:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-22T00:38:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:38:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 638cdf02-b978-469f-a12a-a6548054f4f4 2024-09-22T00:38:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:38:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:38:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:38:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:18.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:38:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 953d8d1a-423b-404a-bfe8-3deffbfa74d4 2024-09-22T00:38:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:38:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:38:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8199a7a1-0f0f-4500-8687-d6ef1ddb91c2 2024-09-22T00:38:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:38:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:38:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:18.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:38:18.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:38:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-22T00:38:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:38:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-22T00:38:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd0 2024-09-22T00:38:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:38:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-22T00:38:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd1 2024-09-22T00:38:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-22T00:38:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:38:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:38:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd2 2024-09-22T00:38:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-22T00:38:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:38:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd3 2024-09-22T00:38:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:38:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd4 2024-09-22T00:38:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-22T00:38:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd5 2024-09-22T00:38:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-22T00:38:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-22T00:38:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:38:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:38:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:38:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 0d381d63-1eb0-4a28-89be-01bd007e4f27 2024-09-22T00:38:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15 2024-09-22T00:38:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:38:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:38:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:38:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:15' 2024-09-22T00:38:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:38:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:38:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:38:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-22 00:38:15' 2024-09-22T00:38:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:38:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:38:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:15' 2024-09-22T00:38:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41+ local status 2024-09-22T00:38:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:38:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:38:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-22T00:38:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:38:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:38:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-22T00:38:18.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:38:18.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:18.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:38:18.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-22T00:38:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:38:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:38:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-22T00:38:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:38:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:19.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:38:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-22T00:38:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:38:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:38:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-22T00:38:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary844563834-6c39-4532-8b94-956e536c455fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:38:15cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:38:15' 2024-09-22T00:38:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:38:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:15' 2024-09-22T00:38:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:38:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:38:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:38:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:38:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:38:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-22T00:38:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:38:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:38:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-22T00:38:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:38:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:38:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:38:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:38:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:38:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-22T00:38:19.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:38:19.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:38:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-22T00:38:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:38:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:38:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:38:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-22T00:38:19.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:38:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:38:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-22T00:38:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd6 2024-09-22T00:38:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:38:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-22T00:38:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd7 2024-09-22T00:38:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-22T00:38:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:38:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd8 2024-09-22T00:38:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-22T00:38:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:38:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd9 2024-09-22T00:38:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-22T00:38:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:38:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=e663780a6567413dee840fc1398da40f 2024-09-22T00:38:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-22T00:38:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=f648ca9cbec83ce919909ca5a99f71f3 2024-09-22T00:38:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-22T00:38:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=4e1e03894d556e3cc6aefe417e7814b9 2024-09-22T00:38:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-22T00:38:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ e663780a6567413dee840fc1398da40f != \e\6\6\3\7\8\0\a\6\5\6\7\4\1\3\d\e\e\8\4\0\f\c\1\3\9\8\d\a\4\0\f ]] 2024-09-22T00:38:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ f648ca9cbec83ce919909ca5a99f71f3 != \f\6\4\8\c\a\9\c\b\e\c\8\3\c\e\9\1\9\9\0\9\c\a\5\a\9\9\f\7\1\f\3 ]] 2024-09-22T00:38:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=2fe4f7c16f9855dd5560b4034aa749f6 2024-09-22T00:38:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-22T00:38:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 4e1e03894d556e3cc6aefe417e7814b9 != \4\e\1\e\0\3\8\9\4\d\5\5\6\e\3\c\c\6\a\e\f\e\4\1\7\e\7\8\1\4\b\9 ]] 2024-09-22T00:38:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=26e14632384bef682f27de3b068a0773 2024-09-22T00:38:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-22T00:38:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 2fe4f7c16f9855dd5560b4034aa749f6 != \2\f\e\4\f\7\c\1\6\f\9\8\5\5\d\d\5\5\6\0\b\4\0\3\4\a\a\7\4\9\f\6 ]] 2024-09-22T00:38:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 26e14632384bef682f27de3b068a0773 != \2\6\e\1\4\6\3\2\3\8\4\b\e\f\6\8\2\f\2\7\d\e\3\b\0\6\8\a\0\7\7\3 ]] 2024-09-22T00:38:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=4e2a816c7e4c201c78b10853253c4a7f 2024-09-22T00:38:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-22T00:38:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=ca9eb77a1d063da78458cb6fa0751a2e 2024-09-22T00:38:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-22T00:38:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 4e2a816c7e4c201c78b10853253c4a7f != \4\e\2\a\8\1\6\c\7\e\4\c\2\0\1\c\7\8\b\1\0\8\5\3\2\5\3\c\4\a\7\f ]] 2024-09-22T00:38:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:38:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 105547 2024-09-22T00:38:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=4848433e769e49cee289dfe98e5f5f4f 2024-09-22T00:38:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-22T00:38:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=157fa3a6efab088615f2d07737a3ca24 2024-09-22T00:38:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-22T00:38:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ ca9eb77a1d063da78458cb6fa0751a2e != \c\a\9\e\b\7\7\a\1\d\0\6\3\d\a\7\8\4\5\8\c\b\6\f\a\0\7\5\1\a\2\e ]] 2024-09-22T00:38:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:38:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 105549 2024-09-22T00:38:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:38:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 105551 2024-09-22T00:38:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:38:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 105552 2024-09-22T00:38:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:38:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 105555 2024-09-22T00:38:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=999a1816f1f8e32b732aa4ca4265e99a 2024-09-22T00:38:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-22T00:38:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 4848433e769e49cee289dfe98e5f5f4f != \4\8\4\8\4\3\3\e\7\6\9\e\4\9\c\e\e\2\8\9\d\f\e\9\8\e\5\f\5\f\4\f ]] 2024-09-22T00:38:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:38:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 105556 2024-09-22T00:38:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:38:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 105558 2024-09-22T00:38:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:38:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 105560 2024-09-22T00:38:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 157fa3a6efab088615f2d07737a3ca24 != \1\5\7\f\a\3\a\6\e\f\a\b\0\8\8\6\1\5\f\2\d\0\7\7\3\7\a\3\c\a\2\4 ]] 2024-09-22T00:38:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:38:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 105562 2024-09-22T00:38:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 999a1816f1f8e32b732aa4ca4265e99a != \9\9\9\a\1\8\1\6\f\1\f\8\e\3\2\b\7\3\2\a\a\4\c\a\4\2\6\5\e\9\9\a ]] 2024-09-22T00:38:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:38:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:38:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-22T00:38:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-22T00:38:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:38:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:38:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-22T00:38:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:38:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:38:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-22T00:38:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:38:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:38:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:38:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:38:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: a360adf9-9dc4-42ed-b481-6c2a0fe2c78a 2024-09-22T00:38:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:38:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":6359859.2,"bytes_per_snapshot":63598592.0,"last_snapshot_bytes":63598592,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"} 2024-09-22T00:38:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:35 2024-09-22T00:38:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:38:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:38:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:38:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:50' 2024-09-22T00:38:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:38:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:38:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:54.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":6359859.2,"bytes_per_snapshot":63598592.0,"last_snapshot_bytes":63598592,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"} 2024-09-22T00:38:54.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:54.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:38:54.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:38:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:35' 2024-09-22T00:38:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-22 00:38:35' 2024-09-22T00:38:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:38:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:38:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:35' 2024-09-22T00:38:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1a360adf9-9dc4-42ed-b481-6c2a0fe2c78aup+replayingreplaying, {"bytes_per_second":6359859.2,"bytes_per_snapshot":63598592.0,"last_snapshot_bytes":63598592,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:50image-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+replayingreplaying, {"bytes_per_second":6501171.2,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:50image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+replayingreplaying, {"bytes_per_second":6378291.2,"bytes_per_snapshot":63782912.0,"last_snapshot_bytes":63782912,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:50image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+replayingreplaying, {"bytes_per_second":6506496.0,"bytes_per_snapshot":65064960.0,"last_snapshot_bytes":65064960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:50image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+replayingreplaying, {"bytes_per_second":6324633.6,"bytes_per_snapshot":63246336.0,"last_snapshot_bytes":63246336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:50image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+replayingreplaying, {"bytes_per_second":6856704.0,"bytes_per_snapshot":68567040.0,"last_snapshot_bytes":68567040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:50image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+replayingreplaying, {"bytes_per_second":6342656.0,"bytes_per_snapshot":63426560.0,"last_snapshot_bytes":63426560,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:50image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+replayingreplaying, {"bytes_per_second":6470451.2,"bytes_per_snapshot":64704512.0,"last_snapshot_bytes":64704512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:50image-primary844563834-6c39-4532-8b94-956e536c455fup+replayingreplaying, {"bytes_per_second":6708428.8,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:50image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+replayingreplaying, {"bytes_per_second":6555238.4,"bytes_per_snapshot":65552384.0,"last_snapshot_bytes":65552384,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:50' 2024-09-22T00:38:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:38:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:35' 2024-09-22T00:38:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:38:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:38:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:38:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:38:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:38:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-22T00:38:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:38:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-22T00:38:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-22T00:38:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:38:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-22T00:38:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:38:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:38:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary1 2024-09-22T00:38:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary1 2024-09-22T00:38:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:38:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-22T00:38:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-22T00:38:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:38:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:38:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-22T00:38:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:38:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:38:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:38:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:38:58.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-22T00:38:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:38:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:38:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:38:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:38:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:38:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:38:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 953d8d1a-423b-404a-bfe8-3deffbfa74d4 2024-09-22T00:38:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:38:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":6378291.2,"bytes_per_snapshot":63782912.0,"last_snapshot_bytes":63782912,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"} 2024-09-22T00:38:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:38:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:35 2024-09-22T00:38:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:38:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:38:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:38:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:38:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:55' 2024-09-22T00:38:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:38:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:38:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:38:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":6378291.2,"bytes_per_snapshot":63782912.0,"last_snapshot_bytes":63782912,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"} 2024-09-22T00:38:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:38:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:38:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:38:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:38:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:38:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:38:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:38:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:38:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:35' 2024-09-22T00:38:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-22 00:38:35' 2024-09-22T00:38:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:38:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:38:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:38:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:35' 2024-09-22T00:38:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:38:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:38:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:38:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK942614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+replayingreplaying, {"bytes_per_second":6501171.2,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:55image-primary2953d8d1a-423b-404a-bfe8-3deffbfa74d4up+replayingreplaying, {"bytes_per_second":6378291.2,"bytes_per_snapshot":63782912.0,"last_snapshot_bytes":63782912,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:55image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+replayingreplaying, {"bytes_per_second":6506496.0,"bytes_per_snapshot":65064960.0,"last_snapshot_bytes":65064960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:55image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+replayingreplaying, {"bytes_per_second":6324633.6,"bytes_per_snapshot":63246336.0,"last_snapshot_bytes":63246336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:55image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+replayingreplaying, {"bytes_per_second":6856704.0,"bytes_per_snapshot":68567040.0,"last_snapshot_bytes":68567040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:55image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+replayingreplaying, {"bytes_per_second":6342656.0,"bytes_per_snapshot":63426560.0,"last_snapshot_bytes":63426560,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:55image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+replayingreplaying, {"bytes_per_second":6470451.2,"bytes_per_snapshot":64704512.0,"last_snapshot_bytes":64704512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:55image-primary844563834-6c39-4532-8b94-956e536c455fup+replayingreplaying, {"bytes_per_second":6708428.8,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:55image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+replayingreplaying, {"bytes_per_second":6555238.4,"bytes_per_snapshot":65552384.0,"last_snapshot_bytes":65552384,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:38:55' 2024-09-22T00:38:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:38:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:38:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:35' 2024-09-22T00:38:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:38:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:38:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:38:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:38:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:38:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:38:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-22T00:38:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:38:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:38:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:38:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-22T00:38:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-22T00:39:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:39:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-22T00:39:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:39:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:39:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:39:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary2 2024-09-22T00:39:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary2 2024-09-22T00:39:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:39:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-22T00:39:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-22T00:39:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:39:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:39:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-22T00:39:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:39:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:39:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:39:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:39:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:39:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-22T00:39:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:39:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:39:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:39:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:39:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:39:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:39:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:39:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: db5a3f2b-5b3d-4185-a630-858f70204abb 2024-09-22T00:39:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:39:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":6506496.0,"bytes_per_snapshot":65064960.0,"last_snapshot_bytes":65064960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"} 2024-09-22T00:39:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:39:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:35 2024-09-22T00:39:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:39:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:39:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:39:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:39:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:39:00' 2024-09-22T00:39:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:39:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:39:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:39:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":6506496.0,"bytes_per_snapshot":65064960.0,"last_snapshot_bytes":65064960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"} 2024-09-22T00:39:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:39:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:39:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:39:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:39:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:39:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:39:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:39:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:39:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:35' 2024-09-22T00:39:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-22 00:38:35' 2024-09-22T00:39:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:39:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:35' 2024-09-22T00:39:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:39:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:39:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:39:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK842614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+replayingreplaying, {"bytes_per_second":6501171.2,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:00image-primary3db5a3f2b-5b3d-4185-a630-858f70204abbup+replayingreplaying, {"bytes_per_second":6506496.0,"bytes_per_snapshot":65064960.0,"last_snapshot_bytes":65064960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:00image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+replayingreplaying, {"bytes_per_second":6324633.6,"bytes_per_snapshot":63246336.0,"last_snapshot_bytes":63246336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:00image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+replayingreplaying, {"bytes_per_second":6856704.0,"bytes_per_snapshot":68567040.0,"last_snapshot_bytes":68567040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:00image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+replayingreplaying, {"bytes_per_second":6342656.0,"bytes_per_snapshot":63426560.0,"last_snapshot_bytes":63426560,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:00image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+replayingreplaying, {"bytes_per_second":6470451.2,"bytes_per_snapshot":64704512.0,"last_snapshot_bytes":64704512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:00image-primary844563834-6c39-4532-8b94-956e536c455fup+replayingreplaying, {"bytes_per_second":6708428.8,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:00image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+replayingreplaying, {"bytes_per_second":6555238.4,"bytes_per_snapshot":65552384.0,"last_snapshot_bytes":65552384,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:00' 2024-09-22T00:39:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:39:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:39:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:35' 2024-09-22T00:39:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:39:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:39:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:39:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:39:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:39:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-22T00:39:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:39:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:39:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-22T00:39:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-22T00:39:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:39:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-22T00:39:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:39:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:39:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:39:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary3 2024-09-22T00:39:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary3 2024-09-22T00:39:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:39:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-22T00:39:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-22T00:39:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:39:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:39:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-22T00:39:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:39:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:39:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:39:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:39:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:39:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-22T00:39:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:39:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:39:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:39:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:39:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:39:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:39:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:39:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 005a21ea-613d-4f8c-8c61-abb670ab6b3f 2024-09-22T00:39:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:39:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":6324633.6,"bytes_per_snapshot":63246336.0,"last_snapshot_bytes":63246336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"} 2024-09-22T00:39:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:39:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:38:35 2024-09-22T00:39:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:39:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:39:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:39:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:39:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:39:00' 2024-09-22T00:39:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:39:04.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:39:04.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:39:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":6324633.6,"bytes_per_snapshot":63246336.0,"last_snapshot_bytes":63246336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"} 2024-09-22T00:39:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:39:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:39:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:39:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:39:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:39:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:39:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:39:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:39:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:35' 2024-09-22T00:39:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-22 00:38:35' 2024-09-22T00:39:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:39:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:38:35' 2024-09-22T00:39:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:39:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:39:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:39:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK742614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+replayingreplaying, {"bytes_per_second":6501171.2,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:00image-primary4005a21ea-613d-4f8c-8c61-abb670ab6b3fup+replayingreplaying, {"bytes_per_second":6324633.6,"bytes_per_snapshot":63246336.0,"last_snapshot_bytes":63246336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:00image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+replayingreplaying, {"bytes_per_second":6856704.0,"bytes_per_snapshot":68567040.0,"last_snapshot_bytes":68567040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:00image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+replayingreplaying, {"bytes_per_second":6342656.0,"bytes_per_snapshot":63426560.0,"last_snapshot_bytes":63426560,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:00image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+replayingreplaying, {"bytes_per_second":6470451.2,"bytes_per_snapshot":64704512.0,"last_snapshot_bytes":64704512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:00image-primary844563834-6c39-4532-8b94-956e536c455fup+replayingreplaying, {"bytes_per_second":6708428.8,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:00image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+replayingreplaying, {"bytes_per_second":6555238.4,"bytes_per_snapshot":65552384.0,"last_snapshot_bytes":65552384,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:38:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:00' 2024-09-22T00:39:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:39:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:39:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:38:35' 2024-09-22T00:39:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:39:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:39:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:39:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:39:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:39:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-22T00:39:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:39:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:39:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-22T00:39:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-22T00:39:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:39:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-22T00:39:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:39:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:39:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:39:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary4 2024-09-22T00:39:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary4 2024-09-22T00:39:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:39:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-22T00:39:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-22T00:39:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:39:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:39:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-22T00:39:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:39:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:39:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:39:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:39:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:39:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-22T00:39:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:39:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:39:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:39:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:39:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:39:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:39:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:39:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 778f8891-97b2-4db6-a75e-3094d57bdd92 2024-09-22T00:39:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:39:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":68567040.0,"last_snapshot_bytes":68567040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"} 2024-09-22T00:39:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:39:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:39:05 2024-09-22T00:39:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:39:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:39:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:39:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:39:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:39:05' 2024-09-22T00:39:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:39:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:39:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:39:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":68567040.0,"last_snapshot_bytes":68567040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"} 2024-09-22T00:39:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:39:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:39:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:39:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:39:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:39:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:39:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:39:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:39:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:39:05' 2024-09-22T00:39:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-22 00:39:05' 2024-09-22T00:39:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:39:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:39:05' 2024-09-22T00:39:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:39:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:39:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:39:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK642614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:05image-primary5778f8891-97b2-4db6-a75e-3094d57bdd92up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":68567040.0,"last_snapshot_bytes":68567040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:05image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63426560.0,"last_snapshot_bytes":63426560,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:05image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64704512.0,"last_snapshot_bytes":64704512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:05image-primary844563834-6c39-4532-8b94-956e536c455fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:05image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65552384.0,"last_snapshot_bytes":65552384,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:05' 2024-09-22T00:39:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:39:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:39:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:39:05' 2024-09-22T00:39:07.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:39:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:39:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:39:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:39:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:39:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-22T00:39:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:39:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:39:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-22T00:39:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-22T00:39:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:39:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-22T00:39:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:39:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:39:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:39:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary5 2024-09-22T00:39:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary5 2024-09-22T00:39:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:39:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-22T00:39:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-22T00:39:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:39:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:39:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-22T00:39:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:39:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:39:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:39:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:39:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:39:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-22T00:39:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:39:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:39:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:39:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:39:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:39:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:39:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:39:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 0d381d63-1eb0-4a28-89be-01bd007e4f27 2024-09-22T00:39:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:39:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63426560.0,"last_snapshot_bytes":63426560,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"} 2024-09-22T00:39:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:39:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:39:05 2024-09-22T00:39:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:39:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:39:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:39:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:39:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:39:10' 2024-09-22T00:39:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:39:10.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:39:10.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:39:10.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63426560.0,"last_snapshot_bytes":63426560,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"} 2024-09-22T00:39:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:39:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:39:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:39:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:39:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:39:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:39:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:39:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:39:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:39:05' 2024-09-22T00:39:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-22 00:39:05' 2024-09-22T00:39:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:39:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:39:05' 2024-09-22T00:39:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:39:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:39:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:39:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK542614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:10image-primary60d381d63-1eb0-4a28-89be-01bd007e4f27up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63426560.0,"last_snapshot_bytes":63426560,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:10image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64704512.0,"last_snapshot_bytes":64704512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:10image-primary844563834-6c39-4532-8b94-956e536c455fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:10image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65552384.0,"last_snapshot_bytes":65552384,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:10' 2024-09-22T00:39:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:39:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:39:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:39:05' 2024-09-22T00:39:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:39:10.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:39:10.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:39:10.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:39:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:39:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-22T00:39:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:39:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:10.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:39:10.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-22T00:39:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-22T00:39:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:39:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-22T00:39:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:39:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:39:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:39:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary6 2024-09-22T00:39:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary6 2024-09-22T00:39:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:39:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-22T00:39:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-22T00:39:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:39:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:39:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-22T00:39:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:39:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:39:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:39:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:39:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:39:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-22T00:39:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:39:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:39:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:39:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:39:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:39:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:39:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:39:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f10a60d7-c2e8-4819-ac0b-9abd9de85975 2024-09-22T00:39:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:39:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64704512.0,"last_snapshot_bytes":64704512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"} 2024-09-22T00:39:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:39:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:39:05 2024-09-22T00:39:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:39:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:39:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:39:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:39:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:39:10' 2024-09-22T00:39:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:39:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:39:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:39:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64704512.0,"last_snapshot_bytes":64704512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"} 2024-09-22T00:39:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:39:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:39:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:39:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:39:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:39:13.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:39:13.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:39:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:39:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:39:05' 2024-09-22T00:39:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-22 00:39:05' 2024-09-22T00:39:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:39:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:39:05' 2024-09-22T00:39:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:39:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:39:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:39:13.797 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:39:13.789+0000 7fab5fe19640 -1 reset not still connected to 0x5623f97bdba0 2024-09-22T00:39:13.797 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:39:13.789+0000 7fab5fe19640 -1 reset not still connected to 0x56240c979860 2024-09-22T00:39:13.797 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:39:13.789+0000 7fab5fe19640 -1 reset not still connected to 0x562411fa04e0 2024-09-22T00:39:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK442614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:10image-primary7f10a60d7-c2e8-4819-ac0b-9abd9de85975up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64704512.0,"last_snapshot_bytes":64704512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:10image-primary844563834-6c39-4532-8b94-956e536c455fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:10image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65552384.0,"last_snapshot_bytes":65552384,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:10' 2024-09-22T00:39:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:39:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:39:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:39:05' 2024-09-22T00:39:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:39:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:39:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:39:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:39:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:39:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-22T00:39:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:39:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:39:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-22T00:39:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-22T00:39:14.941 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:39:14.937+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281450 2024-09-22T00:39:14.942 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:39:14.937+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281520 2024-09-22T00:39:14.942 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:39:14.937+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:39:14.942 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:39:14.937+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T00:39:14.942 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:39:14.937+0000 7f54f2d95640 -1 reset not still connected to 0x56180a15bc70 2024-09-22T00:39:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:39:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-22T00:39:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:39:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:39:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:39:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary7 2024-09-22T00:39:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary7 2024-09-22T00:39:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:39:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-22T00:39:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-22T00:39:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:39:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:39:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-22T00:39:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:39:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:39:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:39:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:39:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:39:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-22T00:39:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:39:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:39:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:39:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:39:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:39:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:39:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:39:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 44563834-6c39-4532-8b94-956e536c455f 2024-09-22T00:39:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:39:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"} 2024-09-22T00:39:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:39:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:39:05 2024-09-22T00:39:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:39:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:39:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:39:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:39:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:39:15' 2024-09-22T00:39:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:39:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:39:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:39:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"} 2024-09-22T00:39:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:39:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:39:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:39:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:39:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:39:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:39:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:39:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:39:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:39:05' 2024-09-22T00:39:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-22 00:39:05' 2024-09-22T00:39:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:39:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:39:05' 2024-09-22T00:39:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:39:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:39:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:39:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:15image-primary844563834-6c39-4532-8b94-956e536c455fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726965498,"remote_snapshot_timestamp":1726965498,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65552384.0,"last_snapshot_bytes":65552384,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:15' 2024-09-22T00:39:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:39:16.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:39:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:39:05' 2024-09-22T00:39:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:39:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:39:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:39:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:39:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:39:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-22T00:39:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:39:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:39:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-22T00:39:17.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-22T00:39:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:39:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-22T00:39:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:39:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:39:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:39:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary8 2024-09-22T00:39:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary8 2024-09-22T00:39:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:39:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-22T00:39:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-22T00:39:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:39:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:39:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-22T00:39:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:39:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:39:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:39:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:39:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:39:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-22T00:39:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:39:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:39:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:39:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:39:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:39:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:39:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:39:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 638cdf02-b978-469f-a12a-a6548054f4f4 2024-09-22T00:39:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:39:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65552384.0,"last_snapshot_bytes":65552384,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"} 2024-09-22T00:39:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:39:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:39:05 2024-09-22T00:39:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:39:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:39:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:39:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:39:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:39:15' 2024-09-22T00:39:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:39:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:39:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:39:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65552384.0,"last_snapshot_bytes":65552384,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"} 2024-09-22T00:39:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:39:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:39:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:39:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:39:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:39:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:39:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:39:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:39:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:39:05' 2024-09-22T00:39:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-22 00:39:05' 2024-09-22T00:39:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:39:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:39:05' 2024-09-22T00:39:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:39:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:39:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:39:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK242614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:15image-primary9638cdf02-b978-469f-a12a-a6548054f4f4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65552384.0,"last_snapshot_bytes":65552384,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:15' 2024-09-22T00:39:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:39:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:39:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:39:05' 2024-09-22T00:39:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:39:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:39:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:39:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:39:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:39:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-22T00:39:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:39:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:39:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-22T00:39:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-22T00:39:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:39:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-22T00:39:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:39:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:39:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:39:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary9 2024-09-22T00:39:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary9 2024-09-22T00:39:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:39:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-22T00:39:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-22T00:39:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:39:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:39:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-22T00:39:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:39:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:39:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:39:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:39:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:39:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-22T00:39:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:39:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:39:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:39:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:39:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:39:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:39:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:39:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8199a7a1-0f0f-4500-8687-d6ef1ddb91c2 2024-09-22T00:39:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:39:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"} 2024-09-22T00:39:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:39:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:39:05 2024-09-22T00:39:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:39:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:39:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:39:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:39:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:39:20' 2024-09-22T00:39:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:39:22.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:39:22.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:39:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"} 2024-09-22T00:39:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:39:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:39:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:39:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:39:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:39:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:39:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:39:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:39:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:39:05' 2024-09-22T00:39:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-22 00:39:05' 2024-09-22T00:39:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:39:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:39:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:39:05' 2024-09-22T00:39:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:39:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:39:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:39:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK142614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary108199a7a1-0f0f-4500-8687-d6ef1ddb91c2up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965499,"remote_snapshot_timestamp":1726965499,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:39:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:39:20' 2024-09-22T00:39:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:39:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:39:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:39:05' 2024-09-22T00:39:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:39:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:39:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:39:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:39:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:39:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-22T00:39:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:39:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:39:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-22T00:39:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-22T00:39:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:39:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-22T00:39:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:39:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:39:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:39:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary10 2024-09-22T00:39:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary10 2024-09-22T00:39:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..10} 2024-09-22T00:39:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS=() 2024-09-22T00:39:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS=() 2024-09-22T00:39:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS=() 2024-09-22T00:39:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS=() 2024-09-22T00:39:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ RET=0 2024-09-22T00:39:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:39:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-22T00:39:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary1 2024-09-22T00:39:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-22T00:39:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:39:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:39:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-22T00:39:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:39:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:39:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:39:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:39:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-22T00:39:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:39:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:39:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-22T00:39:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:39:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-22T00:39:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:39:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-22T00:39:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary1': 2024-09-22T00:39:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:39:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:39:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:39:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 2ff7bd5712eb 2024-09-22T00:39:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.2ff7bd5712eb 2024-09-22T00:39:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:39:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:39:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:39:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:39:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:39:24 2024 2024-09-22T00:39:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:39:24 2024 2024-09-22T00:39:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:39:24 2024 2024-09-22T00:39:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:39:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:39:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 77d20118-e580-49a0-91bd-99b940c4214f 2024-09-22T00:39:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:39:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:39:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:39:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-22T00:39:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd0 2024-09-22T00:39:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:39:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-22T00:39:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:39:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:39:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:39:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: edb70404-05a3-4ba3-a712-a6bde1194798 2024-09-22T00:39:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:39:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:39:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:39:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:39:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:39:26.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:39:26.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary1 2024-09-22T00:39:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-22T00:39:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:39:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary1 2024-09-22T00:39:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:39:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-22T00:39:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:39:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:39:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:39:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-22T00:39:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary2 2024-09-22T00:39:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-22T00:39:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:39:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:39:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:39:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:39:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-22T00:39:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:39:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:39:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:39:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:39:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:39:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary1 2024-09-22T00:39:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-22T00:39:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary1 2024-09-22T00:39:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-22T00:39:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:39:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:39:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-22T00:39:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:39:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-22T00:39:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:39:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-22T00:39:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary2': 2024-09-22T00:39:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:39:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:39:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:39:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 30072fdd90 2024-09-22T00:39:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.30072fdd90 2024-09-22T00:39:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:39:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:39:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:39:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:39:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:39:26 2024 2024-09-22T00:39:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:39:26 2024 2024-09-22T00:39:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:39:26 2024 2024-09-22T00:39:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:39:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:39:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 671129b9-a52a-4c24-af82-2470eec900fc 2024-09-22T00:39:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:39:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:39:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:39:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-22T00:39:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd1 2024-09-22T00:39:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:39:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-22T00:39:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2326 2024-09-22T00:39:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:39:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:39:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:39:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 7bf56847-b08e-4a44-80d5-87eee7c6da7b 2024-09-22T00:39:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:39:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:39:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:39:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:39:27.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:39:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:39:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary2 2024-09-22T00:39:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-22T00:39:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:39:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary2 2024-09-22T00:39:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:39:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-22T00:39:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:39:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:39:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:39:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:39:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:39:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-22T00:39:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:39:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary3 2024-09-22T00:39:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary2 2024-09-22T00:39:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-22T00:39:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:39:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary2 2024-09-22T00:39:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:39:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-22T00:39:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-22T00:39:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:39:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:39:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:39:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:39:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-22T00:39:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:39:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-22T00:39:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:39:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:39:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-22T00:39:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:39:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-22T00:39:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:39:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-22T00:39:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:39:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2328 2024-09-22T00:39:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-22T00:39:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary3': 2024-09-22T00:39:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:39:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:39:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:39:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 3015ac00929f 2024-09-22T00:39:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.3015ac00929f 2024-09-22T00:39:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:39:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:39:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:39:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:39:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:39:27 2024 2024-09-22T00:39:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:39:27 2024 2024-09-22T00:39:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:39:27 2024 2024-09-22T00:39:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:39:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:39:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 171f569f-ad5e-466d-b334-fb68e6575660 2024-09-22T00:39:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:39:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:39:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:39:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-22T00:39:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd2 2024-09-22T00:39:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:39:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-22T00:39:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:39:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:39:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:39:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: f09a2bed-0b23-4960-86f5-75f8abea9db7 2024-09-22T00:39:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:39:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:39:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:39:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:39:28.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:39:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:39:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary3 2024-09-22T00:39:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-22T00:39:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:39:28.327 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary3 2024-09-22T00:39:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:39:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-22T00:39:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:39:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:39:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:39:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:39:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:39:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:39:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary3 2024-09-22T00:39:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-22T00:39:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary4 2024-09-22T00:39:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-22T00:39:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary3 2024-09-22T00:39:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:39:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-22T00:39:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:39:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-22T00:39:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:39:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:39:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:39:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:39:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-22T00:39:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:39:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:39:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-22T00:39:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:39:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-22T00:39:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:39:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-22T00:39:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2330 2024-09-22T00:39:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:39:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-22T00:39:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary4': 2024-09-22T00:39:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:39:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:39:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:39:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 30246bb901c1 2024-09-22T00:39:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.30246bb901c1 2024-09-22T00:39:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:39:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:39:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:39:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:39:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:39:28 2024 2024-09-22T00:39:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:39:28 2024 2024-09-22T00:39:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:39:28 2024 2024-09-22T00:39:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:39:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:39:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: d2519679-3bc6-44ca-83c5-f06f5e7c3616 2024-09-22T00:39:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:39:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:39:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:39:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-22T00:39:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd3 2024-09-22T00:39:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:39:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-22T00:39:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:39:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:39:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:39:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: d1c5f569-6273-48b5-8ce4-4cd96cd43a3e 2024-09-22T00:39:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:39:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:39:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:39:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:39:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:39:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:39:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:29.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary4 2024-09-22T00:39:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-22T00:39:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:39:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary4 2024-09-22T00:39:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:39:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-22T00:39:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:39:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:39:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-22T00:39:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary5 2024-09-22T00:39:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-22T00:39:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:39:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:39:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:39:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:39:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:39:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:39:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-22T00:39:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:39:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:39:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:39:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:39:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-22T00:39:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary4 2024-09-22T00:39:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary4 2024-09-22T00:39:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-22T00:39:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:39:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:39:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-22T00:39:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:39:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-22T00:39:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:39:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-22T00:39:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:39:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:39:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:39:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:39:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2332 2024-09-22T00:39:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:30.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-22T00:39:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary5': 2024-09-22T00:39:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:39:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:39:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:39:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 303166b83b9d 2024-09-22T00:39:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.303166b83b9d 2024-09-22T00:39:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:39:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:39:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:39:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:39:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:39:29 2024 2024-09-22T00:39:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:39:29 2024 2024-09-22T00:39:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:39:29 2024 2024-09-22T00:39:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:39:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:39:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: f522b81f-c203-4fae-beb6-7f02c13b1b76 2024-09-22T00:39:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:39:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:39:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:39:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-22T00:39:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd4 2024-09-22T00:39:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:39:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-22T00:39:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:39:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:39:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:39:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: ae44a789-6514-4884-8189-b7a96cace2c8 2024-09-22T00:39:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:39:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:39:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:39:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:39:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:39:30.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:39:30.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary5 2024-09-22T00:39:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-22T00:39:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:39:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:39:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:39:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:39:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary5 2024-09-22T00:39:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:39:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-22T00:39:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:39:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:39:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:39:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:39:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:39:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:39:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary5 2024-09-22T00:39:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-22T00:39:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary6 2024-09-22T00:39:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-22T00:39:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary5 2024-09-22T00:39:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-22T00:39:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:39:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:39:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-22T00:39:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:39:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:39:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:39:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:39:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-22T00:39:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:39:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:39:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-22T00:39:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:39:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-22T00:39:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:39:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-22T00:39:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2334 2024-09-22T00:39:31.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:39:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2336 2024-09-22T00:39:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:31.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-22T00:39:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary6': 2024-09-22T00:39:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:39:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:39:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:39:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 304637c23692 2024-09-22T00:39:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.304637c23692 2024-09-22T00:39:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:39:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:39:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:39:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:39:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:39:31 2024 2024-09-22T00:39:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:39:31 2024 2024-09-22T00:39:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:39:31 2024 2024-09-22T00:39:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:39:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:39:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 6dd943e6-a8c9-4d29-adf4-b9a4267f8ec0 2024-09-22T00:39:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:39:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:39:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:39:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-22T00:39:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd5 2024-09-22T00:39:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:39:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-22T00:39:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:39:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:39:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:39:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: b0eb509f-20a6-4687-87a5-7f5df87e7006 2024-09-22T00:39:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:39:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:39:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:39:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:39:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:39:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:39:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary6 2024-09-22T00:39:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-22T00:39:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:39:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary6 2024-09-22T00:39:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:39:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-22T00:39:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:39:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:39:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:39:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:39:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:39:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:39:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-22T00:39:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary7 2024-09-22T00:39:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary6 2024-09-22T00:39:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-22T00:39:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:39:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:39:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-22T00:39:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary6 2024-09-22T00:39:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:39:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:39:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-22T00:39:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:39:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:39:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-22T00:39:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:39:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:39:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:39:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:39:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:39:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-22T00:39:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:39:32.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:32.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-22T00:39:32.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:39:32.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-22T00:39:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2335 2024-09-22T00:39:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:32.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2338 2024-09-22T00:39:32.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:39:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-22T00:39:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary7': 2024-09-22T00:39:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:39:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:39:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:39:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 305329de4e45 2024-09-22T00:39:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.305329de4e45 2024-09-22T00:39:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:39:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:39:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:39:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:39:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:39:32 2024 2024-09-22T00:39:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:39:32 2024 2024-09-22T00:39:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:39:32 2024 2024-09-22T00:39:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:39:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:39:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: e3ff7450-7479-43c6-ac3f-42b526002262 2024-09-22T00:39:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:39:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:39:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:39:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-22T00:39:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd6 2024-09-22T00:39:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:39:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-22T00:39:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:39:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:39:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:39:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 576bf709-309f-45e1-888a-a61f6d7f1db0 2024-09-22T00:39:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:39:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:39:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:39:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:39:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:39:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:39:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary7 2024-09-22T00:39:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-22T00:39:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:39:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary7 2024-09-22T00:39:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:39:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-22T00:39:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:39:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:39:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:39:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:39:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-22T00:39:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary7 2024-09-22T00:39:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary8 2024-09-22T00:39:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:39:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-22T00:39:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:39:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:39:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary7 2024-09-22T00:39:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-22T00:39:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:39:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-22T00:39:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:39:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:39:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:39:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:39:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-22T00:39:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:39:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:39:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-22T00:39:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:39:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-22T00:39:33.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:39:33.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-22T00:39:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2339 2024-09-22T00:39:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:39:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:39:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:39:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:39:33.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-22T00:39:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2341 2024-09-22T00:39:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary8': 2024-09-22T00:39:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:39:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:39:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:39:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 30646f4ffb0b 2024-09-22T00:39:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.30646f4ffb0b 2024-09-22T00:39:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:39:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:39:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:39:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:39:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:39:33 2024 2024-09-22T00:39:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:39:33 2024 2024-09-22T00:39:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:39:33 2024 2024-09-22T00:39:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:39:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:39:33.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: d3d29bd1-2a9d-4ae7-baad-00b075a26894 2024-09-22T00:39:33.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:39:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:39:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:39:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-22T00:39:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd7 2024-09-22T00:39:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:39:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-22T00:39:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:39:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:39:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:39:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: d96170c4-782c-4dd6-a935-5cde95bebcfd 2024-09-22T00:39:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:39:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:39:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:39:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:39:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:39:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:39:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary8 2024-09-22T00:39:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-22T00:39:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:39:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary8 2024-09-22T00:39:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:39:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-22T00:39:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:39:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:39:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary8 2024-09-22T00:39:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:39:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-22T00:39:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary9 2024-09-22T00:39:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-22T00:39:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:39:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary8 2024-09-22T00:39:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-22T00:39:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:39:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:39:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:39:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-22T00:39:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:39:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:39:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:39:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:39:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:39:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-22T00:39:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:39:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:39:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-22T00:39:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:39:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-22T00:39:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:39:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-22T00:39:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:39:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:39:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:39:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:39:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:39:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:39:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:39:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2344 2024-09-22T00:39:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-22T00:39:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary9': 2024-09-22T00:39:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:39:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:39:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:39:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 3075e859ae68 2024-09-22T00:39:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.3075e859ae68 2024-09-22T00:39:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:39:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:39:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:39:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:39:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:39:34 2024 2024-09-22T00:39:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:39:34 2024 2024-09-22T00:39:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:39:34 2024 2024-09-22T00:39:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:39:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:39:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 9903594b-6515-4f45-8571-ccb2d516bdbd 2024-09-22T00:39:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:39:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:39:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:39:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-22T00:39:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd8 2024-09-22T00:39:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:39:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-22T00:39:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:39:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:39:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:39:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 26dfb8aa-b8ab-4b9f-a134-2c2c59f454cd 2024-09-22T00:39:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:39:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:39:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:39:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:39:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:39:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:39:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary9 2024-09-22T00:39:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-22T00:39:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:39:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary9 2024-09-22T00:39:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:39:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-22T00:39:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:39:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:39:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:39:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:39:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:39:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary9 2024-09-22T00:39:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:39:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-22T00:39:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary10 2024-09-22T00:39:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-22T00:39:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary9 2024-09-22T00:39:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-22T00:39:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:39:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:39:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-22T00:39:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:39:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:39:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:39:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:39:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:39:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-22T00:39:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:39:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:39:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:39:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-22T00:39:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:39:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:39:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-22T00:39:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:39:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-22T00:39:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2343 2024-09-22T00:39:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:35.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:39:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:39:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:39:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:39:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2348 2024-09-22T00:39:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:39:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:39:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:39:35.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-22T00:39:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary10': 2024-09-22T00:39:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:39:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:39:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:39:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 3087a749af2b 2024-09-22T00:39:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.3087a749af2b 2024-09-22T00:39:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:39:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:39:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:39:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:39:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:39:35 2024 2024-09-22T00:39:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:39:35 2024 2024-09-22T00:39:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:39:35 2024 2024-09-22T00:39:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:39:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:39:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 7ac01973-7feb-4e85-997b-aa8eb80475fa 2024-09-22T00:39:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:39:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:39:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:39:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-22T00:39:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd9 2024-09-22T00:39:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:39:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-22T00:39:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:39:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:39:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:39:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 1476c782-6c7a-4c93-bc86-5305a717abe8 2024-09-22T00:39:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:39:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:39:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:39:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:39:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:39:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:39:36.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:39:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary10 2024-09-22T00:39:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-22T00:39:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:39:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary10 2024-09-22T00:39:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:39:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-22T00:39:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:39:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:39:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:39:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:39:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:39:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:39:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 109253 2024-09-22T00:39:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary10 2024-09-22T00:39:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary10 2024-09-22T00:39:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-22T00:39:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:39:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-22T00:39:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2347 2024-09-22T00:39:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2346 2024-09-22T00:39:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:39:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:39:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:39:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2352 2024-09-22T00:39:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:39:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:39:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:39:37.418 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:39:37.416+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a3630960d0 2024-09-22T00:39:37.418 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:39:37.416+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a364135ba0 2024-09-22T00:39:37.419 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:39:37.416+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a36a516000 2024-09-22T00:39:37.423 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:39:37.420+0000 7f84953c5640 -1 reset not still connected to 0x5636f85f0d00 2024-09-22T00:39:37.423 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:39:37.420+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b332b0 2024-09-22T00:39:37.423 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:39:37.420+0000 7f84953c5640 -1 reset not still connected to 0x5636f8ec80d0 2024-09-22T00:39:37.423 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:39:37.420+0000 7f84953c5640 -1 reset not still connected to 0x5636f9c289c0 2024-09-22T00:39:37.423 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:39:37.420+0000 7f84953c5640 -1 reset not still connected to 0x5637002f5e10 2024-09-22T00:39:37.429 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:39:37.424+0000 7f84953c5640 -1 reset not still connected to 0x56371a23a8f0 2024-09-22T00:39:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2351 2024-09-22T00:39:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2350 2024-09-22T00:39:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:39:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:39:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:39:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2353 2024-09-22T00:39:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:39:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:39:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:39:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2354 2024-09-22T00:39:38.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:39:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:39:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:39:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2355 2024-09-22T00:39:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:39:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:39:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:39:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:39:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:39:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:39:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:39:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:39:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:39:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2357 2024-09-22T00:39:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2356 2024-09-22T00:39:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:39:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:39:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:39:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:39:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:39:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:39:40.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2360 2024-09-22T00:39:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2358 2024-09-22T00:39:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2359 2024-09-22T00:39:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:39:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:39:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:39:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:39:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:39:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:39:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2361 2024-09-22T00:39:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2362 2024-09-22T00:39:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:39:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:39:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:39:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:42.831 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:39:42.825+0000 7f54f2d95640 -1 reset not still connected to 0x5617fe4e4340 2024-09-22T00:39:42.831 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:39:42.825+0000 7f54f2d95640 -1 reset not still connected to 0x561810b77ba0 2024-09-22T00:39:42.831 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:39:42.825+0000 7f54f2d95640 -1 reset not still connected to 0x561828d6f6c0 2024-09-22T00:39:43.504 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:39:43.501+0000 7f54f2d95640 -1 reset not still connected to 0x5617f33328f0 2024-09-22T00:39:43.504 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:39:43.501+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3392ea0 2024-09-22T00:39:43.519 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:39:43.517+0000 7f54f2d95640 -1 reset not still connected to 0x5617f33931e0 2024-09-22T00:39:43.519 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:39:43.517+0000 7f54f2d95640 -1 reset not still connected to 0x5617f9befad0 2024-09-22T00:39:43.519 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:39:43.517+0000 7f54f2d95640 -1 reset not still connected to 0x5618204a5a00 2024-09-22T00:39:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2364 2024-09-22T00:39:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2363 2024-09-22T00:39:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:39:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:39:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:39:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:39:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:39:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:39:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2365 2024-09-22T00:39:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:39:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:39:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:39:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:39:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:39:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:39:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:39:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:39:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:39:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2366 2024-09-22T00:39:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2367 2024-09-22T00:39:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:39:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:39:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:39:45.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:45.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:39:45.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:45.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:45.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:39:45.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:39:45.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:45.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2370 2024-09-22T00:39:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2368 2024-09-22T00:39:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2369 2024-09-22T00:39:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:39:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:39:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:39:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:39:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:39:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:39:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:39:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:39:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:39:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2371 2024-09-22T00:39:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2372 2024-09-22T00:39:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2373 2024-09-22T00:39:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2374 2024-09-22T00:39:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:39:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:39:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:39:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:39:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:39:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:39:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2375 2024-09-22T00:39:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:39:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:39:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:39:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:39:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:39:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:39:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:39:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:39:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:39:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2376 2024-09-22T00:39:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2377 2024-09-22T00:39:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:39:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:39:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:39:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:39:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:39:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:39:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2380 2024-09-22T00:39:51.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2378 2024-09-22T00:39:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:39:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:39:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:39:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2379 2024-09-22T00:39:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:39:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:39:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:39:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:39:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:39:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:39:52.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2382 2024-09-22T00:39:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2381 2024-09-22T00:39:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:52.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:52.730 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:39:52.729+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7cf90bee0 2024-09-22T00:39:52.730 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:39:52.729+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7dcb76340 2024-09-22T00:39:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2384 2024-09-22T00:39:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2383 2024-09-22T00:39:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:39:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:39:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:39:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:39:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:39:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:39:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2385 2024-09-22T00:39:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:39:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:39:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:39:54.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:39:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:39:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:39:54.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:54.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:39:54.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:54.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:39:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:39:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2387 2024-09-22T00:39:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2386 2024-09-22T00:39:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:39:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:39:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:39:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:39:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:39:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:39:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2390 2024-09-22T00:39:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2388 2024-09-22T00:39:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2389 2024-09-22T00:39:56.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:39:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:39:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:39:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:39:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:39:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:39:57.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2391 2024-09-22T00:39:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2392 2024-09-22T00:39:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:39:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:39:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:39:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2394 2024-09-22T00:39:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2393 2024-09-22T00:39:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:39:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:39:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:39:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:39:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:39:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:39:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2395 2024-09-22T00:39:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:39:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:39:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:39:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:39:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:39:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:39:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:39:59.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:39:59.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:39:59.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:39:59.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:39:59.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:39:59.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:40:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2397 2024-09-22T00:40:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2396 2024-09-22T00:40:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:40:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:40:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:40:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:40:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:40:00.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:40:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2400 2024-09-22T00:40:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2398 2024-09-22T00:40:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2399 2024-09-22T00:40:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:40:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:40:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:40:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:40:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:40:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:40:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:40:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:40:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:40:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2402 2024-09-22T00:40:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2401 2024-09-22T00:40:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2404 2024-09-22T00:40:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2403 2024-09-22T00:40:03.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:40:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:40:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:40:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:40:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:40:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:40:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2405 2024-09-22T00:40:04.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:40:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:40:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:40:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:40:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:40:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:40:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:40:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:40:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:40:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2406 2024-09-22T00:40:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2407 2024-09-22T00:40:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:40:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:40:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:40:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:40:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:40:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:40:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2408 2024-09-22T00:40:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2409 2024-09-22T00:40:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:40:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:40:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:40:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2410 2024-09-22T00:40:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:40:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:40:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:40:06.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:06.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:40:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:40:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:40:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2412 2024-09-22T00:40:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2411 2024-09-22T00:40:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2414 2024-09-22T00:40:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2413 2024-09-22T00:40:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:40:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:40:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:40:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:40:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:40:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:40:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2415 2024-09-22T00:40:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:40:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:40:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:40:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:40:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:40:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:40:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:40:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:40:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:40:09.789 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:40:09.788+0000 7f84953c5640 -1 reset not still connected to 0x5636f85f0d00 2024-09-22T00:40:09.796 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:40:09.792+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b332b0 2024-09-22T00:40:09.797 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:40:09.792+0000 7f84953c5640 -1 reset not still connected to 0x5636f8ec80d0 2024-09-22T00:40:09.797 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:40:09.792+0000 7f84953c5640 -1 reset not still connected to 0x5636f9c289c0 2024-09-22T00:40:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2417 2024-09-22T00:40:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2416 2024-09-22T00:40:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:40:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:40:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:40:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:40:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:40:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:40:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2420 2024-09-22T00:40:11.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2418 2024-09-22T00:40:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2419 2024-09-22T00:40:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:40:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:40:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:40:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:40:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:40:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:40:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:40:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:40:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:40:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2421 2024-09-22T00:40:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2422 2024-09-22T00:40:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:12.707 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:40:12.704+0000 7f54f2d95640 -1 reset not still connected to 0x5618204a5a00 2024-09-22T00:40:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2424 2024-09-22T00:40:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2423 2024-09-22T00:40:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:40:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:40:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:40:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:40:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:40:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:40:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2425 2024-09-22T00:40:14.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:40:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:40:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:40:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:40:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:40:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:40:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:40:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:40:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:40:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2426 2024-09-22T00:40:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2427 2024-09-22T00:40:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:40:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:40:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:40:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:40:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:40:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:40:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2430 2024-09-22T00:40:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2428 2024-09-22T00:40:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2429 2024-09-22T00:40:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:40:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:40:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:40:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:40:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:40:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:40:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:40:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:40:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:40:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2431 2024-09-22T00:40:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2432 2024-09-22T00:40:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2433 2024-09-22T00:40:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2434 2024-09-22T00:40:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:40:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:40:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:40:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:40:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:40:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:40:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2435 2024-09-22T00:40:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:40:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:40:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:40:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:40:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:40:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:40:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:40:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:40:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:40:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2437 2024-09-22T00:40:20.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2436 2024-09-22T00:40:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:40:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:40:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:40:20.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:20.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:40:20.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:20.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:20.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:40:20.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:40:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2438 2024-09-22T00:40:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2440 2024-09-22T00:40:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2439 2024-09-22T00:40:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:40:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:40:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:40:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:40:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:40:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:40:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:40:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:40:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:40:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2441 2024-09-22T00:40:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2442 2024-09-22T00:40:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:40:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:40:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:40:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:40:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:40:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:40:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2443 2024-09-22T00:40:23.696 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:40:23.692+0000 7f54f2d95640 -1 reset not still connected to 0x5618204a5a00 2024-09-22T00:40:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2445 2024-09-22T00:40:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2444 2024-09-22T00:40:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:40:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:40:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:40:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:40:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:40:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:40:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:40:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:40:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:40:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2446 2024-09-22T00:40:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2447 2024-09-22T00:40:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:40:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:40:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:40:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:40:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:40:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:40:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2449 2024-09-22T00:40:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2448 2024-09-22T00:40:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2450 2024-09-22T00:40:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:40:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:40:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:40:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2452 2024-09-22T00:40:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2451 2024-09-22T00:40:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:40:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:40:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:40:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:40:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:40:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:40:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2453 2024-09-22T00:40:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:40:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:40:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:40:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:40:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:40:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:40:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2455 2024-09-22T00:40:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2454 2024-09-22T00:40:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:40:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:40:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:40:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:40:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:40:29.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:40:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:40:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:40:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:40:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2457 2024-09-22T00:40:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2456 2024-09-22T00:40:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:40:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:40:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:40:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:40:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:40:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:40:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2459 2024-09-22T00:40:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2458 2024-09-22T00:40:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:40:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:40:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:40:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2460 2024-09-22T00:40:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:40:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:40:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:40:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:40:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:40:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:40:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2462 2024-09-22T00:40:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2461 2024-09-22T00:40:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2464 2024-09-22T00:40:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2463 2024-09-22T00:40:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:40:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:40:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:40:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:40:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:40:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:40:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2465 2024-09-22T00:40:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:40:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:40:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:40:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:40:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:40:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:40:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:40:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:40:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:40:35.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2467 2024-09-22T00:40:35.555 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:40:35.552+0000 7f84953c5640 -1 reset not still connected to 0x5636f85f0d00 2024-09-22T00:40:35.556 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:40:35.552+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b32ea0 2024-09-22T00:40:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2466 2024-09-22T00:40:35.565 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:40:35.560+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b332b0 2024-09-22T00:40:35.565 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:40:35.560+0000 7f84953c5640 -1 reset not still connected to 0x5636f8ec80d0 2024-09-22T00:40:35.565 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:40:35.560+0000 7f84953c5640 -1 reset not still connected to 0x5636f9a67ad0 2024-09-22T00:40:35.566 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:40:35.560+0000 7f84953c5640 -1 reset not still connected to 0x5636f9c289c0 2024-09-22T00:40:35.566 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:40:35.560+0000 7f84953c5640 -1 reset not still connected to 0x5636fa4bf040 2024-09-22T00:40:35.567 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:40:35.560+0000 7f84953c5640 -1 reset not still connected to 0x5636fca38dd0 2024-09-22T00:40:35.568 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:40:35.560+0000 7f84953c5640 -1 reset not still connected to 0x5637002f5e10 2024-09-22T00:40:35.568 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:40:35.560+0000 7f84953c5640 -1 reset not still connected to 0x563704b59ba0 2024-09-22T00:40:35.568 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:40:35.560+0000 7f84953c5640 -1 reset not still connected to 0x56370f167790 2024-09-22T00:40:35.568 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:40:35.560+0000 7f84953c5640 -1 reset not still connected to 0x56371a23a8f0 2024-09-22T00:40:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:40:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:40:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:40:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:40:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:40:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:40:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:40:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:40:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:40:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2470 2024-09-22T00:40:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:40:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:40:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:40:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2469 2024-09-22T00:40:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2468 2024-09-22T00:40:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:40:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:40:37.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:40:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2471 2024-09-22T00:40:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2472 2024-09-22T00:40:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2474 2024-09-22T00:40:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:40:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:40:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:40:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2473 2024-09-22T00:40:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:40:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:40:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:40:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2475 2024-09-22T00:40:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:40:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:40:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:40:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:40:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:40:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:40:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:40:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:40:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:40:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2476 2024-09-22T00:40:40.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2477 2024-09-22T00:40:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:40:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:40:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:40:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:40:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:40:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:40:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2479 2024-09-22T00:40:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2480 2024-09-22T00:40:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2478 2024-09-22T00:40:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:40:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:40:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:40:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:40:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:40:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:40:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:41.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:40:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:40:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:40:42.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2482 2024-09-22T00:40:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2481 2024-09-22T00:40:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2483 2024-09-22T00:40:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2484 2024-09-22T00:40:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:40:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:40:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:40:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:40:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:40:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:40:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2485 2024-09-22T00:40:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:40:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:40:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:40:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:40:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:40:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:40:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:40:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:40:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:40:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2486 2024-09-22T00:40:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2487 2024-09-22T00:40:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:40:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:40:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:40:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:40:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:40:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:40:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2490 2024-09-22T00:40:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2489 2024-09-22T00:40:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:40:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:40:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:40:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2488 2024-09-22T00:40:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:40:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:40:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:40:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:40:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:40:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:40:47.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2492 2024-09-22T00:40:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2491 2024-09-22T00:40:47.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2494 2024-09-22T00:40:48.646 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:40:48.644+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7dcb76340 2024-09-22T00:40:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2493 2024-09-22T00:40:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:40:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:40:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:40:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:40:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:40:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:40:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2495 2024-09-22T00:40:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:40:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:40:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:40:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:40:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:40:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:40:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:40:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:40:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:40:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2497 2024-09-22T00:40:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2496 2024-09-22T00:40:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:40:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:40:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:40:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:40:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:40:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:40:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2500 2024-09-22T00:40:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2499 2024-09-22T00:40:51.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2498 2024-09-22T00:40:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:40:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:40:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:40:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:40:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:40:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:40:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2501 2024-09-22T00:40:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2502 2024-09-22T00:40:52.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:40:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:40:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:40:52.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2503 2024-09-22T00:40:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2504 2024-09-22T00:40:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:40:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:40:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:40:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:40:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:40:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:40:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2505 2024-09-22T00:40:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:40:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:40:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:40:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:40:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:40:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:40:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:40:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:40:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:40:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2507 2024-09-22T00:40:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2506 2024-09-22T00:40:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:40:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:40:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:40:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:40:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:40:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:40:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:55.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2509 2024-09-22T00:40:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2510 2024-09-22T00:40:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:40:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:40:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:40:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2508 2024-09-22T00:40:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:40:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:40:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:40:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:40:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:40:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:40:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2512 2024-09-22T00:40:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2511 2024-09-22T00:40:57.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2514 2024-09-22T00:40:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2513 2024-09-22T00:40:58.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:40:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:40:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:40:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:40:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:40:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:40:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2515 2024-09-22T00:40:59.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:40:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:40:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:40:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:40:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:40:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:40:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:40:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:40:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:40:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:40:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:40:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:40:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:41:00.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2516 2024-09-22T00:41:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2517 2024-09-22T00:41:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:41:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:41:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:41:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:41:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:41:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:41:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2519 2024-09-22T00:41:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2520 2024-09-22T00:41:01.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2518 2024-09-22T00:41:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:41:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:41:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:41:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:41:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:41:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:41:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:41:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:41:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:41:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2522 2024-09-22T00:41:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2521 2024-09-22T00:41:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2523 2024-09-22T00:41:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2524 2024-09-22T00:41:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:41:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:41:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:41:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:41:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:41:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:41:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2525 2024-09-22T00:41:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:41:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:41:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:41:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:41:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:41:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:41:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:41:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:41:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:41:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2527 2024-09-22T00:41:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2526 2024-09-22T00:41:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:41:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:41:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:41:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:41:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:41:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:41:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2530 2024-09-22T00:41:06.594 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:41:06.591+0000 7f84953c5640 -1 reset not still connected to 0x56371a23a8f0 2024-09-22T00:41:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2529 2024-09-22T00:41:06.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2528 2024-09-22T00:41:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:41:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:41:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:41:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:41:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:41:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:41:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:41:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:41:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:41:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2532 2024-09-22T00:41:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2531 2024-09-22T00:41:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:07.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2534 2024-09-22T00:41:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2533 2024-09-22T00:41:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:41:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:41:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:41:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:41:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:41:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:41:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:09.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2535 2024-09-22T00:41:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:41:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:41:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:41:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:41:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:41:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:41:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:41:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:41:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:41:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2536 2024-09-22T00:41:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2537 2024-09-22T00:41:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:41:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:41:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:41:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:41:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:41:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:41:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2539 2024-09-22T00:41:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2538 2024-09-22T00:41:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2541 2024-09-22T00:41:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:41:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:41:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:41:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:41:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:41:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:41:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:41:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:41:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:41:12.594 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:41:12.591+0000 7fab1f496640 -1 reset not still connected to 0x55fa5eb07450 2024-09-22T00:41:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2542 2024-09-22T00:41:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2543 2024-09-22T00:41:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2544 2024-09-22T00:41:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2545 2024-09-22T00:41:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:41:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:41:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:41:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:41:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:41:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:41:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2546 2024-09-22T00:41:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:41:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:41:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:41:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:41:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:41:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:41:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:41:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:41:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:41:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2548 2024-09-22T00:41:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:41:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:41:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:41:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:41:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:41:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:41:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2547 2024-09-22T00:41:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2550 2024-09-22T00:41:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2549 2024-09-22T00:41:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2551 2024-09-22T00:41:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:41:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:41:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:41:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:41:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:41:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:41:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:41:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:41:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:41:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2552 2024-09-22T00:41:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2553 2024-09-22T00:41:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2554 2024-09-22T00:41:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2555 2024-09-22T00:41:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:41:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:41:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:41:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:41:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:41:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:41:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2556 2024-09-22T00:41:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:41:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:41:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:41:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:41:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:41:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:41:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:41:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:41:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:41:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2558 2024-09-22T00:41:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2557 2024-09-22T00:41:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:41:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:41:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:41:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:41:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:41:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:41:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2560 2024-09-22T00:41:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2561 2024-09-22T00:41:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2559 2024-09-22T00:41:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:41:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:41:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:41:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:41:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:41:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:41:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:41:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:41:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:41:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2563 2024-09-22T00:41:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2562 2024-09-22T00:41:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:41:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:41:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:41:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:41:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:41:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:41:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2565 2024-09-22T00:41:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2564 2024-09-22T00:41:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2566 2024-09-22T00:41:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:41:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:41:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:41:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:41:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:41:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:41:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:41:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:41:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:41:25.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2568 2024-09-22T00:41:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2567 2024-09-22T00:41:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:41:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:41:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:41:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:41:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2570 2024-09-22T00:41:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2569 2024-09-22T00:41:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2571 2024-09-22T00:41:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:26.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:41:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:41:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:41:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:41:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:41:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:41:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2573 2024-09-22T00:41:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2572 2024-09-22T00:41:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:41:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:41:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:41:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2574 2024-09-22T00:41:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2575 2024-09-22T00:41:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:41:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:41:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:41:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:41:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:41:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:41:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2576 2024-09-22T00:41:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:41:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:41:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:41:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:41:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:41:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:41:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:41:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:41:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:41:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2578 2024-09-22T00:41:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:41:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:41:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:41:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:41:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:41:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:41:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2577 2024-09-22T00:41:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2580 2024-09-22T00:41:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2579 2024-09-22T00:41:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2581 2024-09-22T00:41:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:41:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:41:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:41:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:41:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:41:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:41:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2583 2024-09-22T00:41:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2582 2024-09-22T00:41:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:41:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:41:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:41:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2585 2024-09-22T00:41:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2584 2024-09-22T00:41:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:41:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:41:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:41:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:41:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:41:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:41:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2586 2024-09-22T00:41:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:41:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:41:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:41:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:41:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:41:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:41:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:41:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:41:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:41:35.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2588 2024-09-22T00:41:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2587 2024-09-22T00:41:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:41:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:41:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:41:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:41:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:41:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:41:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2589 2024-09-22T00:41:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2590 2024-09-22T00:41:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2591 2024-09-22T00:41:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:41:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:41:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:41:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:41:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:41:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:41:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2592 2024-09-22T00:41:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2593 2024-09-22T00:41:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:41:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:41:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:41:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2595 2024-09-22T00:41:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2594 2024-09-22T00:41:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:41:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:41:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:41:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:41:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:41:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:41:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2596 2024-09-22T00:41:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:41:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:41:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:41:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:41:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:41:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:41:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:41:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:41:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:41:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2598 2024-09-22T00:41:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2597 2024-09-22T00:41:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:41:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:41:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:41:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:41:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:41:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:41:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2600 2024-09-22T00:41:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2599 2024-09-22T00:41:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2601 2024-09-22T00:41:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:41:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:41:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:41:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:41:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:41:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:41:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:41:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:41:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:41:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2602 2024-09-22T00:41:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2603 2024-09-22T00:41:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2604 2024-09-22T00:41:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2605 2024-09-22T00:41:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:41:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:41:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:41:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:41:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:41:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:41:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2606 2024-09-22T00:41:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:41:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:41:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:41:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:41:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:41:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:41:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:41:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:41:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:41:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2607 2024-09-22T00:41:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2608 2024-09-22T00:41:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:41:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:41:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:41:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:41:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:41:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:41:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2609 2024-09-22T00:41:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2611 2024-09-22T00:41:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2610 2024-09-22T00:41:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:46.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:46.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:41:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:41:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:41:46.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:46.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:41:46.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:46.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:46.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:41:46.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:41:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2612 2024-09-22T00:41:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:41:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:41:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:41:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2613 2024-09-22T00:41:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2615 2024-09-22T00:41:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2614 2024-09-22T00:41:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:41:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:41:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:41:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:41:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:41:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:41:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2616 2024-09-22T00:41:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:41:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:41:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:41:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:41:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:41:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:41:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:41:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:41:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:41:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2618 2024-09-22T00:41:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2617 2024-09-22T00:41:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:41:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:41:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:41:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:41:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:41:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:41:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2620 2024-09-22T00:41:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2619 2024-09-22T00:41:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2621 2024-09-22T00:41:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:41:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:41:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:41:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:41:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:41:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:41:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:52.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2623 2024-09-22T00:41:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2622 2024-09-22T00:41:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:41:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:41:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:41:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2625 2024-09-22T00:41:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2624 2024-09-22T00:41:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:41:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:41:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:41:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:41:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:41:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:41:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2626 2024-09-22T00:41:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:41:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 109375 2024-09-22T00:41:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:41:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:41:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:41:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:41:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:41:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:41:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2628 2024-09-22T00:41:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2627 2024-09-22T00:41:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:41:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:55.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:55.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:41:55.604 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:41:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:41:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 109507 2024-09-22T00:41:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2629 2024-09-22T00:41:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2630 2024-09-22T00:41:56.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:56.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:41:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 109634 2024-09-22T00:41:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:41:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:41:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:41:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:41:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:41:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:41:57.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2631 2024-09-22T00:41:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2632 2024-09-22T00:41:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:41:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:41:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:41:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:41:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 109782 2024-09-22T00:41:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2633 2024-09-22T00:41:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:41:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:41:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 109929 2024-09-22T00:41:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:41:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:41:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:41:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:41:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:41:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:42:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2634 2024-09-22T00:42:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:42:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:42:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 110077 2024-09-22T00:42:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2635 2024-09-22T00:42:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:42:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:42:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 110225 2024-09-22T00:42:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:42:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 110392 2024-09-22T00:42:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:42:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 110560 2024-09-22T00:42:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:42:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:42:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 109254 2024-09-22T00:42:15.518 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:42:15.514+0000 7f54f2d95640 -1 reset not still connected to 0x5617f2467380 2024-09-22T00:44:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:44:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:44:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:44:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 109376 2024-09-22T00:44:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:44:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:44:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:44:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 109508 2024-09-22T00:44:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:44:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:44:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:44:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 109635 2024-09-22T00:44:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:44:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:44:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:44:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 109783 2024-09-22T00:44:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:44:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:44:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:44:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 109930 2024-09-22T00:44:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:44:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:44:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:44:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 110078 2024-09-22T00:44:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:44:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:44:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:44:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 110226 2024-09-22T00:44:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:44:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:44:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:44:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 110393 2024-09-22T00:44:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:44:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:44:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:44:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 110561 2024-09-22T00:44:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:44:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:44:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:44:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:44:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:44:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:44:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 1 2024-09-22T00:44:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd0 2024-09-22T00:44:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:44:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary1 2024-09-22T00:44:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary1 2024-09-22T00:44:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:44:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 2 2024-09-22T00:44:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary1 2024-09-22T00:44:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:44:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd1 2024-09-22T00:44:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary2 2024-09-22T00:44:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary2 2024-09-22T00:44:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:44:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary2 2024-09-22T00:44:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:44:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:44:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 3 2024-09-22T00:44:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 4 2024-09-22T00:44:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd2 2024-09-22T00:44:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:44:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary3 2024-09-22T00:44:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary3 2024-09-22T00:44:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:44:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:44:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary3 2024-09-22T00:44:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd3 2024-09-22T00:44:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary4 2024-09-22T00:44:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary4 2024-09-22T00:44:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:44:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary4 2024-09-22T00:44:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:44:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:44:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 5 2024-09-22T00:44:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:44:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd4 2024-09-22T00:44:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary5 2024-09-22T00:44:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary5 2024-09-22T00:44:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:44:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:44:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 6 2024-09-22T00:44:36.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary5 2024-09-22T00:44:36.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd5 2024-09-22T00:44:36.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary6 2024-09-22T00:44:36.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary6 2024-09-22T00:44:36.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:44:36.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary6 2024-09-22T00:44:36.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 7 2024-09-22T00:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd6 2024-09-22T00:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary7 2024-09-22T00:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary7 2024-09-22T00:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary7 2024-09-22T00:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 116698 2024-09-22T00:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 10 2024-09-22T00:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd9 2024-09-22T00:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary10 2024-09-22T00:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary10 2024-09-22T00:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary10 2024-09-22T00:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 9 2024-09-22T00:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd8 2024-09-22T00:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary9 2024-09-22T00:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary9 2024-09-22T00:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary9 2024-09-22T00:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 8 2024-09-22T00:44:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd7 2024-09-22T00:44:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary8 2024-09-22T00:44:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary8 2024-09-22T00:44:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:44:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary8 2024-09-22T00:44:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-22T00:44:36.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:44:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:44:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-22T00:44:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-22T00:44:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:44:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:44:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-22T00:44:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-22T00:44:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:44:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:44:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-22T00:44:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-22T00:44:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:44:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:44:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:44:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-22T00:44:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-22T00:44:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:44:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-22T00:45:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=91efb79625fed7a67703f8f893f831d1 2024-09-22T00:45:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-22T00:45:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=706f58035f2f7cb88cb029a8a8267768 2024-09-22T00:45:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-22T00:45:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-22T00:45:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:45:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-22T00:45:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-22T00:45:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:45:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-22T00:45:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=003de508e43c99af53c262f4a4268faf 2024-09-22T00:45:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-22T00:45:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=a2c6e31309f0ad43f7696859276869df 2024-09-22T00:45:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-22T00:45:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-22T00:45:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:45:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-22T00:45:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-22T00:45:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:45:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-22T00:45:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=b69453142dc20dc3c0b6458128a7f5b1 2024-09-22T00:45:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-22T00:45:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=b41e6b448940f89c95f546f19a18b889 2024-09-22T00:45:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-22T00:45:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=d93b066969daeb47ba33c666ea9cb014 2024-09-22T00:45:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-22T00:45:09.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-22T00:45:09.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:09.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:09.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:45:09.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-22T00:45:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=23255a7183eac07c3d0a6659b56f8549 2024-09-22T00:45:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-22T00:45:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-22T00:45:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:45:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-22T00:45:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-22T00:45:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:45:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-22T00:45:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-22T00:45:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:45:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-22T00:45:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:45:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:45:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-22T00:45:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:45:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-22T00:45:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:45:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=8b0d8f5f2aa1f91ef3a4f718449efacb 2024-09-22T00:45:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-22T00:45:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-22T00:45:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:45:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-22T00:45:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=d4ceb6ac263caa7cbe2bad7850668fdc 2024-09-22T00:45:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-22T00:45:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-22T00:45:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:45:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-22T00:45:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:45:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-22T00:45:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:45:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:11.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-22T00:45:11.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:11.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:11.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:45:11.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:11.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:11.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:11.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:11.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:11.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:45:11.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:45:11.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:45:11.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:11.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:45:11.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:45:11.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:45:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-22T00:45:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:45:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-22T00:45:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:45:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-22T00:45:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:45:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-22T00:45:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:45:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-22T00:45:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:45:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:45:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 7ac01973-7feb-4e85-997b-aa8eb80475fa 2024-09-22T00:45:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10 2024-09-22T00:45:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:45:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10' 2024-09-22T00:45:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:45:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6dd943e6-a8c9-4d29-adf4-b9a4267f8ec0 2024-09-22T00:45:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10 2024-09-22T00:45:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:45:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10' 2024-09-22T00:45:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:45:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:45:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:11.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:10' 2024-09-22T00:45:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-22 00:45:10' 2024-09-22T00:45:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:45:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:10' 2024-09-22T00:45:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:10' 2024-09-22T00:45:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-22 00:45:10' 2024-09-22T00:45:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:45:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:10' 2024-09-22T00:45:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3020800.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965710,"remote_snapshot_timestamp":1726965710,"replay_state":"idle"}2024-09-22 00:45:05image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:10image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4378624.0,"last_snapshot_bytes":126976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965711,"remote_snapshot_timestamp":1726965711,"replay_state":"idle"}2024-09-22 00:45:05image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1683456.0,"last_snapshot_bytes":1626112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965712,"remote_snapshot_timestamp":1726965712,"replay_state":"idle"}2024-09-22 00:45:05image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":923648.0,"last_snapshot_bytes":180224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965714,"remote_snapshot_timestamp":1726965714,"replay_state":"idle"}2024-09-22 00:45:05image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:11image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:10image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965717,"remote_snapshot_timestamp":1726965717,"replay_state":"idle"}2024-09-22 00:45:05image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3588096.0,"last_snapshot_bytes":5345280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965718,"remote_snapshot_timestamp":1726965718,"replay_state":"idle"}2024-09-22 00:45:05image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1695744.0,"last_snapshot_bytes":1671168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965719,"remote_snapshot_timestamp":1726965719,"replay_state":"idle"}2024-09-22 00:45:05' 2024-09-22T00:45:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:45:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:10' 2024-09-22T00:45:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:45:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3020800.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965710,"remote_snapshot_timestamp":1726965710,"replay_state":"idle"}2024-09-22 00:45:05image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:10image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4378624.0,"last_snapshot_bytes":126976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965711,"remote_snapshot_timestamp":1726965711,"replay_state":"idle"}2024-09-22 00:45:05image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1683456.0,"last_snapshot_bytes":1626112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965712,"remote_snapshot_timestamp":1726965712,"replay_state":"idle"}2024-09-22 00:45:05image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":923648.0,"last_snapshot_bytes":180224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965714,"remote_snapshot_timestamp":1726965714,"replay_state":"idle"}2024-09-22 00:45:05image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:11image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:10image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965717,"remote_snapshot_timestamp":1726965717,"replay_state":"idle"}2024-09-22 00:45:05image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3588096.0,"last_snapshot_bytes":5345280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965718,"remote_snapshot_timestamp":1726965718,"replay_state":"idle"}2024-09-22 00:45:05image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1695744.0,"last_snapshot_bytes":1671168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965719,"remote_snapshot_timestamp":1726965719,"replay_state":"idle"}2024-09-22 00:45:05' 2024-09-22T00:45:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:45:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:10' 2024-09-22T00:45:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:45:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-22T00:45:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:45:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-22T00:45:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:45:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:45:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:45:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-22T00:45:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:45:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:45:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-22T00:45:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:45:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:45:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-22T00:45:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:45:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-22T00:45:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:45:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:45:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-22T00:45:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:45:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-22T00:45:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:45:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-22T00:45:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:45:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-22T00:45:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:45:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:45:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:45:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:45:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:45:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 171f569f-ad5e-466d-b334-fb68e6575660 2024-09-22T00:45:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10 2024-09-22T00:45:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:45:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:45:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:12' 2024-09-22T00:45:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:45:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: d2519679-3bc6-44ca-83c5-f06f5e7c3616 2024-09-22T00:45:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10 2024-09-22T00:45:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:45:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopping_replay 2024-09-22T00:45:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:45:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:12' 2024-09-22T00:45:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:45:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:45:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:45:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 671129b9-a52a-4c24-af82-2470eec900fc 2024-09-22T00:45:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10 2024-09-22T00:45:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:45:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:45:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:12' 2024-09-22T00:45:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:12.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:45:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:12.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:12.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:45:12.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 77d20118-e580-49a0-91bd-99b940c4214f 2024-09-22T00:45:12.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:12.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:12.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10 2024-09-22T00:45:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:45:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:45:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:12' 2024-09-22T00:45:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:45:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:10' 2024-09-22T00:45:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-22 00:45:10' 2024-09-22T00:45:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:45:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:10' 2024-09-22T00:45:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:45:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f522b81f-c203-4fae-beb6-7f02c13b1b76 2024-09-22T00:45:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10 2024-09-22T00:45:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:45:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:45:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:11' 2024-09-22T00:45:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:45:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:12.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:12.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:10' 2024-09-22T00:45:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-22 00:45:10' 2024-09-22T00:45:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:45:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:10' 2024-09-22T00:45:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:10' 2024-09-22T00:45:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-22 00:45:10' 2024-09-22T00:45:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:45:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:10' 2024-09-22T00:45:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:10' 2024-09-22T00:45:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-22 00:45:10' 2024-09-22T00:45:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:45:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:10' 2024-09-22T00:45:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:10' 2024-09-22T00:45:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-22 00:45:10' 2024-09-22T00:45:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:45:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:10' 2024-09-22T00:45:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:12.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-22T00:45:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:45:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:12.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-22T00:45:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-22T00:45:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:45:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-22T00:45:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:10image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:11image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:10image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965717,"remote_snapshot_timestamp":1726965717,"replay_state":"idle"}2024-09-22 00:45:05image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3588096.0,"last_snapshot_bytes":5345280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965718,"remote_snapshot_timestamp":1726965718,"replay_state":"idle"}2024-09-22 00:45:05image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1695744.0,"last_snapshot_bytes":1671168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965719,"remote_snapshot_timestamp":1726965719,"replay_state":"idle"}2024-09-22 00:45:05' 2024-09-22T00:45:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:45:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:10' 2024-09-22T00:45:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:45:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:10image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:11image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:10image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965717,"remote_snapshot_timestamp":1726965717,"replay_state":"idle"}2024-09-22 00:45:05image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3588096.0,"last_snapshot_bytes":5345280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965718,"remote_snapshot_timestamp":1726965718,"replay_state":"idle"}2024-09-22 00:45:05image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1695744.0,"last_snapshot_bytes":1671168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965719,"remote_snapshot_timestamp":1726965719,"replay_state":"idle"}2024-09-22 00:45:05' 2024-09-22T00:45:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:45:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:10image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:11image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:10image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965717,"remote_snapshot_timestamp":1726965717,"replay_state":"idle"}2024-09-22 00:45:05image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3588096.0,"last_snapshot_bytes":5345280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965718,"remote_snapshot_timestamp":1726965718,"replay_state":"idle"}2024-09-22 00:45:05image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1695744.0,"last_snapshot_bytes":1671168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965719,"remote_snapshot_timestamp":1726965719,"replay_state":"idle"}2024-09-22 00:45:05' 2024-09-22T00:45:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:45:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-22T00:45:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:45:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:10image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:11image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:10image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965717,"remote_snapshot_timestamp":1726965717,"replay_state":"idle"}2024-09-22 00:45:05image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3588096.0,"last_snapshot_bytes":5345280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965718,"remote_snapshot_timestamp":1726965718,"replay_state":"idle"}2024-09-22 00:45:05image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1695744.0,"last_snapshot_bytes":1671168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965719,"remote_snapshot_timestamp":1726965719,"replay_state":"idle"}2024-09-22 00:45:05' 2024-09-22T00:45:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:45:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:10' 2024-09-22T00:45:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:10' 2024-09-22T00:45:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:45:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:45:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:10' 2024-09-22T00:45:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:45:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-22T00:45:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:45:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-22T00:45:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:45:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-22T00:45:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:45:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:10image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:12image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:45:11image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:10image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965717,"remote_snapshot_timestamp":1726965717,"replay_state":"idle"}2024-09-22 00:45:05image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3588096.0,"last_snapshot_bytes":5345280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965718,"remote_snapshot_timestamp":1726965718,"replay_state":"idle"}2024-09-22 00:45:05image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+stoppedlocal image is primary42614264mirror.0smithi0602024-09-22 00:45:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1695744.0,"last_snapshot_bytes":1671168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965719,"remote_snapshot_timestamp":1726965719,"replay_state":"idle"}2024-09-22 00:45:05' 2024-09-22T00:45:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:45:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6dd943e6-a8c9-4d29-adf4-b9a4267f8ec0 2024-09-22T00:45:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10 2024-09-22T00:45:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:45:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10' 2024-09-22T00:45:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:45:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 7ac01973-7feb-4e85-997b-aa8eb80475fa 2024-09-22T00:45:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10 2024-09-22T00:45:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:45:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10' 2024-09-22T00:45:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:45:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:45:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:45:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:10' 2024-09-22T00:45:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:45:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:10' 2024-09-22T00:45:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-22 00:45:10' 2024-09-22T00:45:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:45:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:10' 2024-09-22T00:45:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:10' 2024-09-22T00:45:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-22 00:45:10' 2024-09-22T00:45:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:45:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:10' 2024-09-22T00:45:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-22T00:45:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:45:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965717,"remote_snapshot_timestamp":1726965717,"replay_state":"idle"}42634269mirror.0smithi0602024-09-22 00:45:05cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3588096.0,"last_snapshot_bytes":5345280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965718,"remote_snapshot_timestamp":1726965718,"replay_state":"idle"}42634269mirror.0smithi0602024-09-22 00:45:05cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1695744.0,"last_snapshot_bytes":1671168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965719,"remote_snapshot_timestamp":1726965719,"replay_state":"idle"}42634269mirror.0smithi0602024-09-22 00:45:05cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10' 2024-09-22T00:45:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:45:13.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:10' 2024-09-22T00:45:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:45:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING7342634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965717,"remote_snapshot_timestamp":1726965717,"replay_state":"idle"}42634269mirror.0smithi0602024-09-22 00:45:05cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3588096.0,"last_snapshot_bytes":5345280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965718,"remote_snapshot_timestamp":1726965718,"replay_state":"idle"}42634269mirror.0smithi0602024-09-22 00:45:05cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1695744.0,"last_snapshot_bytes":1671168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965719,"remote_snapshot_timestamp":1726965719,"replay_state":"idle"}42634269mirror.0smithi0602024-09-22 00:45:05cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10' 2024-09-22T00:45:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:45:13.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:10' 2024-09-22T00:45:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-22T00:45:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:45:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:45:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-22T00:45:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:45:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-22T00:45:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:45:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:45:13.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-22T00:45:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-22T00:45:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:45:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:45:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-22T00:45:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:45:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-22T00:45:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:45:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:45:13.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:13.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:45:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:45:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:45:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:45:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9903594b-6515-4f45-8571-ccb2d516bdbd 2024-09-22T00:45:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:45:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:45:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10 2024-09-22T00:45:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:45:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:45:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:13' 2024-09-22T00:45:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:45:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:45:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:45:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:45:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:45:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:45:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-22T00:45:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-22T00:45:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:45:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: d3d29bd1-2a9d-4ae7-baad-00b075a26894 2024-09-22T00:45:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:45:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:45:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10 2024-09-22T00:45:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:45:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:45:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:13' 2024-09-22T00:45:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:45:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e3ff7450-7479-43c6-ac3f-42b526002262 2024-09-22T00:45:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:45:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:45:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10 2024-09-22T00:45:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:45:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:45:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:13' 2024-09-22T00:45:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:45:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:45:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:45:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 2 2024-09-22T00:45:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd0 2024-09-22T00:45:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-22T00:45:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:45:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd1 2024-09-22T00:45:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-22T00:45:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:45:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-22T00:45:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:45:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-22T00:45:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-22T00:45:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:45:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-22T00:45:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:45:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-22T00:45:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-22T00:45:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-22T00:45:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:45:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-22T00:45:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-22T00:45:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:45:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-22T00:45:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:45:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: d2519679-3bc6-44ca-83c5-f06f5e7c3616 2024-09-22T00:45:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:45:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:12 2024-09-22T00:45:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:45:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10' 2024-09-22T00:45:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:45:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:13.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:13.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:13.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:12' 2024-09-22T00:45:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-22 00:45:12' 2024-09-22T00:45:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:45:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:12' 2024-09-22T00:45:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:45:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 671129b9-a52a-4c24-af82-2470eec900fc 2024-09-22T00:45:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:45:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:12 2024-09-22T00:45:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:45:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10' 2024-09-22T00:45:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:45:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:45:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f522b81f-c203-4fae-beb6-7f02c13b1b76 2024-09-22T00:45:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:45:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:11 2024-09-22T00:45:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:45:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10' 2024-09-22T00:45:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:45:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 77d20118-e580-49a0-91bd-99b940c4214f 2024-09-22T00:45:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:45:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:12 2024-09-22T00:45:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:45:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10' 2024-09-22T00:45:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:13.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:45:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:12' 2024-09-22T00:45:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-22 00:45:12' 2024-09-22T00:45:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:45:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:12' 2024-09-22T00:45:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:12' 2024-09-22T00:45:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-22 00:45:12' 2024-09-22T00:45:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:45:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:12' 2024-09-22T00:45:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:45:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:45:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 171f569f-ad5e-466d-b334-fb68e6575660 2024-09-22T00:45:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:45:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:12 2024-09-22T00:45:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:45:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10' 2024-09-22T00:45:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:45:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:11' 2024-09-22T00:45:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-22 00:45:11' 2024-09-22T00:45:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:45:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:11' 2024-09-22T00:45:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:12' 2024-09-22T00:45:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-22 00:45:12' 2024-09-22T00:45:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:45:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:12' 2024-09-22T00:45:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:13cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:13cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:13cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10' 2024-09-22T00:45:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:45:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:12' 2024-09-22T00:45:14.190 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:45:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:13cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:13cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:13cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10' 2024-09-22T00:45:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:45:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:12' 2024-09-22T00:45:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:45:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-22T00:45:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:45:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:45:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-22T00:45:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-22T00:45:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:45:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:45:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-22T00:45:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:13cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:13cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:13cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10' 2024-09-22T00:45:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:45:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:12' 2024-09-22T00:45:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:45:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-22T00:45:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:45:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:45:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-22T00:45:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:13cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:13cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:13cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10' 2024-09-22T00:45:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:45:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:12cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:10image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:13cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:13cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:45:13cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+stoppedlocal image is primary2024-09-22 00:45:10' 2024-09-22T00:45:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:45:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:12' 2024-09-22T00:45:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:45:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:11' 2024-09-22T00:45:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:45:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-22T00:45:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:45:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:45:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-22T00:45:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:14.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:14.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:14.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:14.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-22T00:45:14.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:14.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:14.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:45:14.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:45:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-22T00:45:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:45:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:45:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:45:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:45:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:45:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:45:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:45:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-22T00:45:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:45:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:45:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:45:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:45:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-22T00:45:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-22T00:45:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:45:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:45:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-22T00:45:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:45:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:45:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-22T00:45:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-22T00:45:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:45:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:45:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-22T00:45:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:45:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:45:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-22T00:45:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:45:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:45:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd3 2024-09-22T00:45:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-22T00:45:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd2 2024-09-22T00:45:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:45:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:45:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-22T00:45:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd4 2024-09-22T00:45:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-22T00:45:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd5 2024-09-22T00:45:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:45:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-22T00:45:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:45:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9903594b-6515-4f45-8571-ccb2d516bdbd 2024-09-22T00:45:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:45:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:45:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10 2024-09-22T00:45:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:45:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:45:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:13' 2024-09-22T00:45:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd6 2024-09-22T00:45:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:45:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-09-22T00:45:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:45:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:45:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-22T00:45:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:45:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: d3d29bd1-2a9d-4ae7-baad-00b075a26894 2024-09-22T00:45:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:45:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:45:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10 2024-09-22T00:45:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:45:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:45:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:13' 2024-09-22T00:45:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:45:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-09-22T00:45:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:45:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e3ff7450-7479-43c6-ac3f-42b526002262 2024-09-22T00:45:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:45:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:45:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:10 2024-09-22T00:45:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:45:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:45:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:13' 2024-09-22T00:45:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 1 2024-09-22T00:45:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 4 2024-09-22T00:45:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-22T00:45:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:45:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:45:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-22T00:45:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:45:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-22T00:45:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:45:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:45:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:45:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:45:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9903594b-6515-4f45-8571-ccb2d516bdbd 2024-09-22T00:45:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:15 2024-09-22T00:45:19.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:45:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:15' 2024-09-22T00:45:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:45:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:45:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: d3d29bd1-2a9d-4ae7-baad-00b075a26894 2024-09-22T00:45:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:15 2024-09-22T00:45:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:45:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:15' 2024-09-22T00:45:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:15' 2024-09-22T00:45:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-22 00:45:15' 2024-09-22T00:45:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:45:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:15' 2024-09-22T00:45:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:45:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e3ff7450-7479-43c6-ac3f-42b526002262 2024-09-22T00:45:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:15 2024-09-22T00:45:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:45:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:15' 2024-09-22T00:45:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:45:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:45:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:19.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:19.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:15' 2024-09-22T00:45:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-22 00:45:15' 2024-09-22T00:45:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:45:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:15' 2024-09-22T00:45:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:15' 2024-09-22T00:45:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-22 00:45:15' 2024-09-22T00:45:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:45:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:15' 2024-09-22T00:45:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING3742614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:15image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:15image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:15' 2024-09-22T00:45:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:45:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:15' 2024-09-22T00:45:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:45:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-22T00:45:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:45:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING3742614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:15image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:15image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:15' 2024-09-22T00:45:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:45:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:15' 2024-09-22T00:45:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:45:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-22T00:45:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:45:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING3742614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:15image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:15image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:15image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:45:15cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image is not primary2024-09-22 00:45:15' 2024-09-22T00:45:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:45:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:15' 2024-09-22T00:45:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:45:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-22T00:45:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:45:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-22T00:45:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:45:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-22T00:45:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-22T00:45:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:45:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-22T00:45:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-22T00:45:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:45:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:20.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-22T00:45:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:45:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e3ff7450-7479-43c6-ac3f-42b526002262 2024-09-22T00:45:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:20 2024-09-22T00:45:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:45:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:20' 2024-09-22T00:45:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:45:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: d3d29bd1-2a9d-4ae7-baad-00b075a26894 2024-09-22T00:45:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:20 2024-09-22T00:45:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:45:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:20' 2024-09-22T00:45:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:45:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9903594b-6515-4f45-8571-ccb2d516bdbd 2024-09-22T00:45:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:20 2024-09-22T00:45:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:45:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:45:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:45:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:20' 2024-09-22T00:45:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:45:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:45:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:45:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:45:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:45:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:21.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:21.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:21.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:21.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:21.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:21.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:20' 2024-09-22T00:45:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-22 00:45:20' 2024-09-22T00:45:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:45:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:20' 2024-09-22T00:45:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:20' 2024-09-22T00:45:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-22 00:45:20' 2024-09-22T00:45:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:45:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:20' 2024-09-22T00:45:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:20' 2024-09-22T00:45:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-22 00:45:20' 2024-09-22T00:45:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:45:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:45:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:20' 2024-09-22T00:45:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING3742634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}2024-09-22 00:45:15image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}2024-09-22 00:45:15image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}2024-09-22 00:45:15image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}2024-09-22 00:45:15image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}2024-09-22 00:45:15image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}2024-09-22 00:45:15image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}2024-09-22 00:45:15image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:20image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:20image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:20' 2024-09-22T00:45:21.480 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING3742634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}2024-09-22 00:45:15image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}2024-09-22 00:45:15image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}2024-09-22 00:45:15image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}2024-09-22 00:45:15image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}2024-09-22 00:45:15image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}2024-09-22 00:45:15image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}2024-09-22 00:45:15image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:20image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:20image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:20' 2024-09-22T00:45:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:45:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:45:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:20' 2024-09-22T00:45:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:45:21.511 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:20' 2024-09-22T00:45:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING3742634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}2024-09-22 00:45:15image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}2024-09-22 00:45:15image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}2024-09-22 00:45:15image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}2024-09-22 00:45:15image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}2024-09-22 00:45:15image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}2024-09-22 00:45:15image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+stoppedlocal image is primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}2024-09-22 00:45:15image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:20image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:20image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+unknownremote image is not primary42634269mirror.0smithi0602024-09-22 00:45:20cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:45:20' 2024-09-22T00:45:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-22T00:45:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:45:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:45:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-22T00:45:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:45:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:45:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:20' 2024-09-22T00:45:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-22T00:45:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:45:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:45:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-22T00:45:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:45:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:45:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:45:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:45:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:45:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-22T00:45:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:21.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:45:21.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:45:21.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-22T00:45:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:45:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:45:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:45:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:45:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:45:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:45:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:45:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-22T00:45:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-22T00:45:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:45:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:45:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-22T00:45:22.714 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:45:22.705+0000 7f84953c5640 -1 reset not still connected to 0x56370ed8dc70 2024-09-22T00:45:22.715 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:45:22.705+0000 7f84953c5640 -1 reset not still connected to 0x563713905790 2024-09-22T00:45:22.720 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:45:22.713+0000 7f84953c5640 -1 reset not still connected to 0x563713905930 2024-09-22T00:45:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd7 2024-09-22T00:45:22.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-22T00:45:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:45:22.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd8 2024-09-22T00:45:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-22T00:45:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd9 2024-09-22T00:45:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:45:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-22T00:45:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:45:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=91efb79625fed7a67703f8f893f831d1 2024-09-22T00:45:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-22T00:45:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=706f58035f2f7cb88cb029a8a8267768 2024-09-22T00:45:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-22T00:45:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 706f58035f2f7cb88cb029a8a8267768 != \7\0\6\f\5\8\0\3\5\f\2\f\7\c\b\8\8\c\b\0\2\9\a\8\a\8\2\6\7\7\6\8 ]] 2024-09-22T00:45:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 91efb79625fed7a67703f8f893f831d1 != \9\1\e\f\b\7\9\6\2\5\f\e\d\7\a\6\7\7\0\3\f\8\f\8\9\3\f\8\3\1\d\1 ]] 2024-09-22T00:45:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=b69453142dc20dc3c0b6458128a7f5b1 2024-09-22T00:45:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-22T00:45:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ b69453142dc20dc3c0b6458128a7f5b1 != \b\6\9\4\5\3\1\4\2\d\c\2\0\d\c\3\c\0\b\6\4\5\8\1\2\8\a\7\f\5\b\1 ]] 2024-09-22T00:45:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=a2c6e31309f0ad43f7696859276869df 2024-09-22T00:45:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-22T00:45:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ a2c6e31309f0ad43f7696859276869df != \a\2\c\6\e\3\1\3\0\9\f\0\a\d\4\3\f\7\6\9\6\8\5\9\2\7\6\8\6\9\d\f ]] 2024-09-22T00:45:47.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=b41e6b448940f89c95f546f19a18b889 2024-09-22T00:45:47.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-22T00:45:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ b41e6b448940f89c95f546f19a18b889 != \b\4\1\e\6\b\4\4\8\9\4\0\f\8\9\c\9\5\f\5\4\6\f\1\9\a\1\8\b\8\8\9 ]] 2024-09-22T00:45:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=d93b066969daeb47ba33c666ea9cb014 2024-09-22T00:45:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-22T00:45:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ d93b066969daeb47ba33c666ea9cb014 != \d\9\3\b\0\6\6\9\6\9\d\a\e\b\4\7\b\a\3\3\c\6\6\6\e\a\9\c\b\0\1\4 ]] 2024-09-22T00:45:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:45:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 116699 2024-09-22T00:45:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=003de508e43c99af53c262f4a4268faf 2024-09-22T00:45:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-22T00:45:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 003de508e43c99af53c262f4a4268faf != \0\0\3\d\e\5\0\8\e\4\3\c\9\9\a\f\5\3\c\2\6\2\f\4\a\4\2\6\8\f\a\f ]] 2024-09-22T00:45:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:45:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 116701 2024-09-22T00:45:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:45:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 116703 2024-09-22T00:45:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:45:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 116704 2024-09-22T00:45:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:45:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 116707 2024-09-22T00:45:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:45:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 116709 2024-09-22T00:45:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=23255a7183eac07c3d0a6659b56f8549 2024-09-22T00:45:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-22T00:45:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=8b0d8f5f2aa1f91ef3a4f718449efacb 2024-09-22T00:45:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-22T00:45:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=d4ceb6ac263caa7cbe2bad7850668fdc 2024-09-22T00:45:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-22T00:45:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 23255a7183eac07c3d0a6659b56f8549 != \2\3\2\5\5\a\7\1\8\3\e\a\c\0\7\c\3\d\0\a\6\6\5\9\b\5\6\f\8\5\4\9 ]] 2024-09-22T00:45:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 8b0d8f5f2aa1f91ef3a4f718449efacb != \8\b\0\d\8\f\5\f\2\a\a\1\f\9\1\e\f\3\a\4\f\7\1\8\4\4\9\e\f\a\c\b ]] 2024-09-22T00:45:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ d4ceb6ac263caa7cbe2bad7850668fdc != \d\4\c\e\b\6\a\c\2\6\3\c\a\a\7\c\b\e\2\b\a\d\7\8\5\0\6\6\8\f\d\c ]] 2024-09-22T00:45:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:45:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 116711 2024-09-22T00:45:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:45:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 116712 2024-09-22T00:45:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:45:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 116714 2024-09-22T00:45:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:45:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:45:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-22T00:45:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-22T00:45:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:45:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:45:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-22T00:45:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:45:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:45:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-22T00:45:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:45:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:45:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:45:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:45:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 77d20118-e580-49a0-91bd-99b940c4214f 2024-09-22T00:45:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:45:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":6573260.8,"bytes_per_snapshot":65732608.0,"last_snapshot_bytes":65732608,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"} 2024-09-22T00:45:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:35 2024-09-22T00:45:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:45:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:45:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:45:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:50' 2024-09-22T00:45:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:45:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:45:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":6573260.8,"bytes_per_snapshot":65732608.0,"last_snapshot_bytes":65732608,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"} 2024-09-22T00:45:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:45:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:45:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:35' 2024-09-22T00:45:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-22 00:45:35' 2024-09-22T00:45:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:45:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:45:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:35' 2024-09-22T00:45:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary177d20118-e580-49a0-91bd-99b940c4214fup+replayingreplaying, {"bytes_per_second":6573260.8,"bytes_per_snapshot":65732608.0,"last_snapshot_bytes":65732608,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:50image-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+replayingreplaying, {"bytes_per_second":6618316.8,"bytes_per_snapshot":66183168.0,"last_snapshot_bytes":66183168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:50image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+replayingreplaying, {"bytes_per_second":6533939.2,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:50image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+replayingreplaying, {"bytes_per_second":7197900.8,"bytes_per_snapshot":71979008.0,"last_snapshot_bytes":71979008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:50image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+replayingreplaying, {"bytes_per_second":7216742.4,"bytes_per_snapshot":72167424.0,"last_snapshot_bytes":72167424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:50image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+replayingreplaying, {"bytes_per_second":6171443.2,"bytes_per_snapshot":61714432.0,"last_snapshot_bytes":61714432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:50image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+replayingreplaying, {"bytes_per_second":6659276.8,"bytes_per_snapshot":66592768.0,"last_snapshot_bytes":66592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:50image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66699264.0,"last_snapshot_bytes":66699264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:50image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:50image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71905280.0,"last_snapshot_bytes":71905280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:50' 2024-09-22T00:45:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:45:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:35' 2024-09-22T00:45:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:45:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:45:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:45:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:45:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:45:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-22T00:45:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:45:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-22T00:45:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-22T00:45:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:45:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-22T00:45:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:45:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:45:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary1 2024-09-22T00:45:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary1 2024-09-22T00:45:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:45:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-22T00:45:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-22T00:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-22T00:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:45:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-22T00:45:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:45:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:45:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:45:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:45:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 671129b9-a52a-4c24-af82-2470eec900fc 2024-09-22T00:45:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:45:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":6533939.2,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"} 2024-09-22T00:45:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:45:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:35 2024-09-22T00:45:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:45:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:45:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:45:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:45:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:55' 2024-09-22T00:45:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:45:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:45:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:45:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":6533939.2,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"} 2024-09-22T00:45:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:45:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:45:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:45:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:45:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:45:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:45:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:45:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:45:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:35' 2024-09-22T00:45:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-22 00:45:35' 2024-09-22T00:45:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:45:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:45:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:35' 2024-09-22T00:45:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:45:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:45:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:45:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK942614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+replayingreplaying, {"bytes_per_second":6618316.8,"bytes_per_snapshot":66183168.0,"last_snapshot_bytes":66183168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:55image-primary2671129b9-a52a-4c24-af82-2470eec900fcup+replayingreplaying, {"bytes_per_second":6533939.2,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:55image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+replayingreplaying, {"bytes_per_second":7197900.8,"bytes_per_snapshot":71979008.0,"last_snapshot_bytes":71979008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:55image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+replayingreplaying, {"bytes_per_second":7216742.4,"bytes_per_snapshot":72167424.0,"last_snapshot_bytes":72167424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:55image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+replayingreplaying, {"bytes_per_second":6171443.2,"bytes_per_snapshot":61714432.0,"last_snapshot_bytes":61714432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:55image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+replayingreplaying, {"bytes_per_second":6659276.8,"bytes_per_snapshot":66592768.0,"last_snapshot_bytes":66592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:55image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66699264.0,"last_snapshot_bytes":66699264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:55image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:55image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71905280.0,"last_snapshot_bytes":71905280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:45:55' 2024-09-22T00:45:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:45:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:45:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:35' 2024-09-22T00:45:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:45:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:45:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:45:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:45:58.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:45:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-22T00:45:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:45:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:45:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-22T00:45:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-22T00:45:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:45:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-22T00:45:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:45:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:45:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary2 2024-09-22T00:45:59.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:45:59.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary2 2024-09-22T00:45:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:45:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-22T00:45:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-22T00:45:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:45:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:45:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-22T00:45:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:45:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:45:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:45:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:45:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:45:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:45:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:45:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:46:00.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-22T00:46:00.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:00.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:00.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:46:00.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:00.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:46:00.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:46:00.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:46:00.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:46:00.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:46:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:46:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 171f569f-ad5e-466d-b334-fb68e6575660 2024-09-22T00:46:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:46:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":7197900.8,"bytes_per_snapshot":71979008.0,"last_snapshot_bytes":71979008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"} 2024-09-22T00:46:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:46:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:35 2024-09-22T00:46:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:46:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:46:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:46:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:46:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:46:00' 2024-09-22T00:46:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:46:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:46:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:46:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":7197900.8,"bytes_per_snapshot":71979008.0,"last_snapshot_bytes":71979008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"} 2024-09-22T00:46:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:46:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:46:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:46:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:46:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:46:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:46:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:46:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:46:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:35' 2024-09-22T00:46:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-22 00:45:35' 2024-09-22T00:46:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:46:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:35' 2024-09-22T00:46:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:46:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:46:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:46:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK842614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+replayingreplaying, {"bytes_per_second":6618316.8,"bytes_per_snapshot":66183168.0,"last_snapshot_bytes":66183168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:00image-primary3171f569f-ad5e-466d-b334-fb68e6575660up+replayingreplaying, {"bytes_per_second":7197900.8,"bytes_per_snapshot":71979008.0,"last_snapshot_bytes":71979008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:00image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+replayingreplaying, {"bytes_per_second":7216742.4,"bytes_per_snapshot":72167424.0,"last_snapshot_bytes":72167424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:00image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+replayingreplaying, {"bytes_per_second":6171443.2,"bytes_per_snapshot":61714432.0,"last_snapshot_bytes":61714432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:00image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+replayingreplaying, {"bytes_per_second":6659276.8,"bytes_per_snapshot":66592768.0,"last_snapshot_bytes":66592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:00image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66699264.0,"last_snapshot_bytes":66699264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:00image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:00image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71905280.0,"last_snapshot_bytes":71905280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:00' 2024-09-22T00:46:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:46:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:46:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:35' 2024-09-22T00:46:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:46:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:46:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:46:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:46:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:46:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-22T00:46:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:46:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:46:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-22T00:46:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-22T00:46:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:46:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-22T00:46:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:46:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:46:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:46:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary3 2024-09-22T00:46:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary3 2024-09-22T00:46:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:46:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-22T00:46:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-22T00:46:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:46:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:46:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-22T00:46:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:46:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:46:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:46:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:46:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:46:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-22T00:46:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:46:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:46:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:46:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:46:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:46:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:46:03.945 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:03.935+0000 7f54f2d95640 -1 reset not still connected to 0x5617f2467380 2024-09-22T00:46:03.946 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:03.939+0000 7f54f2d95640 -1 reset not still connected to 0x5617f324f380 2024-09-22T00:46:03.946 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:03.939+0000 7f54f2d95640 -1 reset not still connected to 0x5617f324f450 2024-09-22T00:46:03.946 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:03.939+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281450 2024-09-22T00:46:03.946 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:03.939+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281520 2024-09-22T00:46:03.946 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:03.939+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:46:03.946 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:03.939+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T00:46:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:46:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: d2519679-3bc6-44ca-83c5-f06f5e7c3616 2024-09-22T00:46:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:46:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":7216742.4,"bytes_per_snapshot":72167424.0,"last_snapshot_bytes":72167424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"} 2024-09-22T00:46:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:46:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:45:35 2024-09-22T00:46:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:46:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:46:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:46:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:46:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:46:00' 2024-09-22T00:46:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:46:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:46:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:46:03.949 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:03.947+0000 7f54f2d95640 -1 reset not still connected to 0x5617f5068ea0 2024-09-22T00:46:03.949 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:03.947+0000 7f54f2d95640 -1 reset not still connected to 0x5617fb743860 2024-09-22T00:46:03.949 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:03.947+0000 7f54f2d95640 -1 reset not still connected to 0x5617fbb7b040 2024-09-22T00:46:03.949 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:03.947+0000 7f54f2d95640 -1 reset not still connected to 0x5618232540d0 2024-09-22T00:46:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":7216742.4,"bytes_per_snapshot":72167424.0,"last_snapshot_bytes":72167424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"} 2024-09-22T00:46:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:46:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:46:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:46:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:46:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:46:03.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:46:03.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:46:03.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:46:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:35' 2024-09-22T00:46:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-22 00:45:35' 2024-09-22T00:46:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:46:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:45:35' 2024-09-22T00:46:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:46:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:46:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:46:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK742614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+replayingreplaying, {"bytes_per_second":6618316.8,"bytes_per_snapshot":66183168.0,"last_snapshot_bytes":66183168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:00image-primary4d2519679-3bc6-44ca-83c5-f06f5e7c3616up+replayingreplaying, {"bytes_per_second":7216742.4,"bytes_per_snapshot":72167424.0,"last_snapshot_bytes":72167424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:00image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+replayingreplaying, {"bytes_per_second":6171443.2,"bytes_per_snapshot":61714432.0,"last_snapshot_bytes":61714432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:00image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+replayingreplaying, {"bytes_per_second":6659276.8,"bytes_per_snapshot":66592768.0,"last_snapshot_bytes":66592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:00image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66699264.0,"last_snapshot_bytes":66699264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:00image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:00image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71905280.0,"last_snapshot_bytes":71905280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:45:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:00' 2024-09-22T00:46:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:46:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:46:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:45:35' 2024-09-22T00:46:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:46:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:46:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:46:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:46:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:46:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-22T00:46:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:46:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:46:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-22T00:46:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-22T00:46:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:46:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-22T00:46:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:46:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:46:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:46:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary4 2024-09-22T00:46:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary4 2024-09-22T00:46:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:46:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-22T00:46:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-22T00:46:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:46:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:46:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-22T00:46:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:46:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:46:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:46:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:46:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:46:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-22T00:46:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:46:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:46:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:46:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:46:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:46:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:46:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:46:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: f522b81f-c203-4fae-beb6-7f02c13b1b76 2024-09-22T00:46:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:46:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61714432.0,"last_snapshot_bytes":61714432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"} 2024-09-22T00:46:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:46:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:46:05 2024-09-22T00:46:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:46:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:46:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:46:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:46:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:46:05' 2024-09-22T00:46:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:46:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:46:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:46:07.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61714432.0,"last_snapshot_bytes":61714432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"} 2024-09-22T00:46:07.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:46:07.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:46:07.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:46:07.137 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:46:07.138 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:46:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:46:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:46:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:46:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:46:05' 2024-09-22T00:46:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-22 00:46:05' 2024-09-22T00:46:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:46:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:46:05' 2024-09-22T00:46:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:46:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:46:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:46:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK642614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66183168.0,"last_snapshot_bytes":66183168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:05image-primary5f522b81f-c203-4fae-beb6-7f02c13b1b76up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61714432.0,"last_snapshot_bytes":61714432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965915,"remote_snapshot_timestamp":1726965915,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:05image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66592768.0,"last_snapshot_bytes":66592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:05image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+replayingreplaying, {"bytes_per_second":2223308.8,"bytes_per_snapshot":66699264.0,"last_snapshot_bytes":66699264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:05image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+replayingreplaying, {"bytes_per_second":2109576.53,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:05image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+replayingreplaying, {"bytes_per_second":2396842.67,"bytes_per_snapshot":71905280.0,"last_snapshot_bytes":71905280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:05' 2024-09-22T00:46:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:46:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:46:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:46:05' 2024-09-22T00:46:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:46:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:46:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:46:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:46:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:46:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-22T00:46:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:46:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:46:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-22T00:46:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-22T00:46:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:46:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-22T00:46:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:46:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:46:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:46:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary5 2024-09-22T00:46:08.759 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:08.755+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281520 2024-09-22T00:46:08.759 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:08.755+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:46:08.759 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:08.755+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T00:46:08.764 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:08.759+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:46:08.765 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:08.759+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T00:46:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary5 2024-09-22T00:46:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:46:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-22T00:46:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-22T00:46:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:46:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:46:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-22T00:46:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:46:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:46:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:46:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:46:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:46:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-22T00:46:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:46:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:46:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:46:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:46:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:46:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:46:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:46:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6dd943e6-a8c9-4d29-adf4-b9a4267f8ec0 2024-09-22T00:46:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:46:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66592768.0,"last_snapshot_bytes":66592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"} 2024-09-22T00:46:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:46:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:46:05 2024-09-22T00:46:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:46:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:46:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:46:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:46:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:46:05' 2024-09-22T00:46:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:46:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:46:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:46:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66592768.0,"last_snapshot_bytes":66592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"} 2024-09-22T00:46:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:46:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:46:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:46:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:46:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:46:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:46:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:46:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:46:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:46:05' 2024-09-22T00:46:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-22 00:46:05' 2024-09-22T00:46:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:46:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:46:05' 2024-09-22T00:46:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:46:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:46:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:46:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK542614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66183168.0,"last_snapshot_bytes":66183168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:05image-primary66dd943e6-a8c9-4d29-adf4-b9a4267f8ec0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66592768.0,"last_snapshot_bytes":66592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:05image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+replayingreplaying, {"bytes_per_second":2223308.8,"bytes_per_snapshot":66699264.0,"last_snapshot_bytes":66699264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:05image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+replayingreplaying, {"bytes_per_second":2109576.53,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:05image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+replayingreplaying, {"bytes_per_second":2396842.67,"bytes_per_snapshot":71905280.0,"last_snapshot_bytes":71905280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:05' 2024-09-22T00:46:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:46:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:46:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:46:05' 2024-09-22T00:46:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:46:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:46:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:46:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:46:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:46:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-22T00:46:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:46:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:46:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-22T00:46:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-22T00:46:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:46:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-22T00:46:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:46:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:46:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:46:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary6 2024-09-22T00:46:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary6 2024-09-22T00:46:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:46:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-22T00:46:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-22T00:46:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:46:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:46:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-22T00:46:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:46:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:46:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:46:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:46:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:46:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-22T00:46:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:46:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:46:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:46:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:46:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:46:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:46:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:46:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e3ff7450-7479-43c6-ac3f-42b526002262 2024-09-22T00:46:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:46:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2223308.8,"bytes_per_snapshot":66699264.0,"last_snapshot_bytes":66699264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"} 2024-09-22T00:46:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:46:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:46:05 2024-09-22T00:46:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:46:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:46:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:46:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:46:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:46:10' 2024-09-22T00:46:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:46:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:46:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:46:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2223308.8,"bytes_per_snapshot":66699264.0,"last_snapshot_bytes":66699264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"} 2024-09-22T00:46:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:46:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:46:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:46:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:46:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:46:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:46:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:46:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:46:12.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:46:05' 2024-09-22T00:46:12.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-22 00:46:05' 2024-09-22T00:46:12.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:12.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:46:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:46:05' 2024-09-22T00:46:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:46:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:46:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:46:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK442614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66183168.0,"last_snapshot_bytes":66183168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:10image-primary7e3ff7450-7479-43c6-ac3f-42b526002262up+replayingreplaying, {"bytes_per_second":2223308.8,"bytes_per_snapshot":66699264.0,"last_snapshot_bytes":66699264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:10image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+replayingreplaying, {"bytes_per_second":2109576.53,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:10image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+replayingreplaying, {"bytes_per_second":2396842.67,"bytes_per_snapshot":71905280.0,"last_snapshot_bytes":71905280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:10' 2024-09-22T00:46:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:46:13.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:46:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:46:05' 2024-09-22T00:46:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:46:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:46:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:46:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:46:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:46:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-22T00:46:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:46:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:46:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-22T00:46:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-22T00:46:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:46:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-22T00:46:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:46:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:46:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:46:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary7 2024-09-22T00:46:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary7 2024-09-22T00:46:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:46:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-22T00:46:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-22T00:46:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:46:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:46:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-22T00:46:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:46:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:46:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:46:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:46:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:46:14.987 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:46:14.983+0000 7fab5fe19640 -1 reset not still connected to 0x56240a9891e0 2024-09-22T00:46:14.988 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:46:14.983+0000 7fab5fe19640 -1 reset not still connected to 0x5624377aec30 2024-09-22T00:46:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-22T00:46:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:46:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:46:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:46:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:46:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:46:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:46:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:46:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: d3d29bd1-2a9d-4ae7-baad-00b075a26894 2024-09-22T00:46:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:46:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2109576.53,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"} 2024-09-22T00:46:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:46:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:46:05 2024-09-22T00:46:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:46:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:46:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:46:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:46:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:46:15' 2024-09-22T00:46:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:46:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:46:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:46:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2109576.53,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"} 2024-09-22T00:46:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:46:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:46:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:46:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:46:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:46:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:46:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:46:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:46:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:46:05' 2024-09-22T00:46:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-22 00:46:05' 2024-09-22T00:46:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:46:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:46:05' 2024-09-22T00:46:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:46:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:46:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:46:16.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66183168.0,"last_snapshot_bytes":66183168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:15image-primary8d3d29bd1-2a9d-4ae7-baad-00b075a26894up+replayingreplaying, {"bytes_per_second":2109576.53,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:15image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+replayingreplaying, {"bytes_per_second":2396842.67,"bytes_per_snapshot":71905280.0,"last_snapshot_bytes":71905280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:15' 2024-09-22T00:46:16.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:46:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:46:16.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:46:05' 2024-09-22T00:46:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:46:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:46:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:46:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:46:16.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:46:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-22T00:46:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:46:16.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:16.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:46:16.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-22T00:46:16.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-22T00:46:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:46:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-22T00:46:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:46:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:46:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:46:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary8 2024-09-22T00:46:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary8 2024-09-22T00:46:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:46:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-22T00:46:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-22T00:46:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:46:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:46:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-22T00:46:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:46:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:46:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:46:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:46:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:46:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-22T00:46:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:46:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:46:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:46:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:46:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:46:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:46:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:46:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 9903594b-6515-4f45-8571-ccb2d516bdbd 2024-09-22T00:46:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:46:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":2396842.67,"bytes_per_snapshot":71905280.0,"last_snapshot_bytes":71905280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"} 2024-09-22T00:46:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:46:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:46:05 2024-09-22T00:46:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:46:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:46:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:46:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:46:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:46:15' 2024-09-22T00:46:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:46:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:46:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:46:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":2396842.67,"bytes_per_snapshot":71905280.0,"last_snapshot_bytes":71905280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"} 2024-09-22T00:46:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:46:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:46:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:46:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:46:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:46:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:46:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:46:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:46:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:46:05' 2024-09-22T00:46:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-22 00:46:05' 2024-09-22T00:46:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:46:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:46:05' 2024-09-22T00:46:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:46:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:46:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:46:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK242614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66183168.0,"last_snapshot_bytes":66183168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:15image-primary99903594b-6515-4f45-8571-ccb2d516bdbdup+replayingreplaying, {"bytes_per_second":2396842.67,"bytes_per_snapshot":71905280.0,"last_snapshot_bytes":71905280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965922,"remote_snapshot_timestamp":1726965922,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:15' 2024-09-22T00:46:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:46:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:46:19.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:46:05' 2024-09-22T00:46:19.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:46:19.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:46:19.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:46:19.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:46:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:46:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-22T00:46:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:46:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:46:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-22T00:46:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-22T00:46:19.469 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:46:19.467+0000 7fab5fe19640 -1 reset not still connected to 0x5623fd37cc30 2024-09-22T00:46:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:46:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-22T00:46:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:46:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:46:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:46:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary9 2024-09-22T00:46:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary9 2024-09-22T00:46:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:46:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-22T00:46:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-22T00:46:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:46:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:46:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-22T00:46:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:46:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:46:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:46:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:46:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:46:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-22T00:46:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:46:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:46:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:46:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:46:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:46:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:46:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:46:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 7ac01973-7feb-4e85-997b-aa8eb80475fa 2024-09-22T00:46:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:46:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66183168.0,"last_snapshot_bytes":66183168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"} 2024-09-22T00:46:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:46:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:46:05 2024-09-22T00:46:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:46:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:46:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:46:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:46:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:46:20' 2024-09-22T00:46:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:46:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:46:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:46:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66183168.0,"last_snapshot_bytes":66183168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"} 2024-09-22T00:46:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:46:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:46:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:46:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:46:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:46:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:46:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:46:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:46:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:46:05' 2024-09-22T00:46:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-22 00:46:05' 2024-09-22T00:46:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:46:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:46:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:46:05' 2024-09-22T00:46:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:46:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:46:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:46:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK142614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary107ac01973-7feb-4e85-997b-aa8eb80475faup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66183168.0,"last_snapshot_bytes":66183168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726965913,"remote_snapshot_timestamp":1726965913,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:46:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:46:20' 2024-09-22T00:46:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:46:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:46:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:46:05' 2024-09-22T00:46:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:46:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:46:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:46:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:46:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:46:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-22T00:46:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:46:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:46:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-22T00:46:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-22T00:46:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:46:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-22T00:46:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:46:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:46:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:46:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary10 2024-09-22T00:46:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary10 2024-09-22T00:46:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..10} 2024-09-22T00:46:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS=() 2024-09-22T00:46:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS=() 2024-09-22T00:46:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS=() 2024-09-22T00:46:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS=() 2024-09-22T00:46:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ RET=0 2024-09-22T00:46:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:46:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-22T00:46:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary1 2024-09-22T00:46:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-22T00:46:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:46:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:46:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-22T00:46:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:46:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:46:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:46:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:46:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-22T00:46:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:46:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:46:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-22T00:46:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:46:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-22T00:46:24.563 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:24.559+0000 7f54f2d95640 -1 reset not still connected to 0x5617f324f380 2024-09-22T00:46:24.566 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:24.563+0000 7f54f2d95640 -1 reset not still connected to 0x5617f324f450 2024-09-22T00:46:24.569 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:24.567+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281450 2024-09-22T00:46:24.569 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:24.567+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281520 2024-09-22T00:46:24.569 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:24.567+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:46:24.569 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:24.567+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T00:46:24.569 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:24.567+0000 7f54f2d95640 -1 reset not still connected to 0x561815880340 2024-09-22T00:46:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:46:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-22T00:46:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary1': 2024-09-22T00:46:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:46:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:46:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:46:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 345f598f9916 2024-09-22T00:46:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.345f598f9916 2024-09-22T00:46:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:46:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:46:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:46:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:46:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:46:23 2024 2024-09-22T00:46:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:46:23 2024 2024-09-22T00:46:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:46:23 2024 2024-09-22T00:46:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:46:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:46:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 747e5399-1569-4c2b-9ca7-3e020ff9d82c 2024-09-22T00:46:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:46:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:46:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:46:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-22T00:46:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd0 2024-09-22T00:46:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:46:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-22T00:46:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:46:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:46:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:46:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: c4304c49-83b5-4165-8f54-2dd813f8239e 2024-09-22T00:46:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:46:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:46:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:46:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:46:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:46:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:46:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary1 2024-09-22T00:46:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-22T00:46:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:46:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary1 2024-09-22T00:46:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:46:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-22T00:46:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:46:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:46:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-22T00:46:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary2 2024-09-22T00:46:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-22T00:46:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:46:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:46:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-22T00:46:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:46:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:46:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:46:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:46:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary1 2024-09-22T00:46:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-22T00:46:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:46:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:46:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:46:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:46:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary1 2024-09-22T00:46:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-22T00:46:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:46:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:46:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-22T00:46:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:46:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-22T00:46:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2657 2024-09-22T00:46:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:46:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:25.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-22T00:46:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary2': 2024-09-22T00:46:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:46:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:46:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:46:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 346eb4c0b112 2024-09-22T00:46:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.346eb4c0b112 2024-09-22T00:46:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:46:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:46:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:46:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:46:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:46:25 2024 2024-09-22T00:46:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:46:25 2024 2024-09-22T00:46:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:46:25 2024 2024-09-22T00:46:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:46:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:46:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 8685aec3-047e-4aad-924b-f779ecacae9d 2024-09-22T00:46:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:46:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:46:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:46:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-22T00:46:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd1 2024-09-22T00:46:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:46:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-22T00:46:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:46:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:46:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:46:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: f9866c36-b2d1-40aa-9482-c1df998a971e 2024-09-22T00:46:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:46:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:46:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:46:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:46:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:46:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:46:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-22T00:46:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:46:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary2 2024-09-22T00:46:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-22T00:46:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:46:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary2 2024-09-22T00:46:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:46:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-22T00:46:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:46:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:46:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:46:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:46:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:46:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:46:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-22T00:46:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary2 2024-09-22T00:46:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary3 2024-09-22T00:46:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-22T00:46:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:46:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary2 2024-09-22T00:46:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:46:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-22T00:46:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-22T00:46:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:46:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:46:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:46:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:46:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-22T00:46:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:46:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:46:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-22T00:46:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:46:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-22T00:46:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:46:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-22T00:46:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2659 2024-09-22T00:46:26.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:46:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-22T00:46:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary3': 2024-09-22T00:46:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:46:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:46:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:46:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 347eb4953103 2024-09-22T00:46:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.347eb4953103 2024-09-22T00:46:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:46:26 2024 2024-09-22T00:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:46:26 2024 2024-09-22T00:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:46:26 2024 2024-09-22T00:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 951a197c-f95a-4cd5-b5ed-190987539508 2024-09-22T00:46:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:46:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:46:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:46:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-22T00:46:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd2 2024-09-22T00:46:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:46:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-22T00:46:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:46:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:46:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:46:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: deef7473-7bdd-4fa9-b7ca-5af1972107e2 2024-09-22T00:46:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:46:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:46:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:46:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:46:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:46:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:46:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary3 2024-09-22T00:46:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-22T00:46:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:46:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary3 2024-09-22T00:46:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:46:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-22T00:46:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:46:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:46:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:46:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary3 2024-09-22T00:46:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-22T00:46:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:46:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary4 2024-09-22T00:46:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-22T00:46:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:46:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:46:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary3 2024-09-22T00:46:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:46:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-22T00:46:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:46:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-22T00:46:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:46:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:46:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:46:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:46:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-22T00:46:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:46:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:46:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-22T00:46:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:46:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-22T00:46:27.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:46:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-22T00:46:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:46:27.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2661 2024-09-22T00:46:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-22T00:46:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary4': 2024-09-22T00:46:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:46:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:46:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:46:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 348c5c26ef36 2024-09-22T00:46:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.348c5c26ef36 2024-09-22T00:46:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:46:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:46:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:46:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:46:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:46:27 2024 2024-09-22T00:46:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:46:27 2024 2024-09-22T00:46:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:46:27 2024 2024-09-22T00:46:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:46:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:46:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 6a888d77-180b-47c1-9533-64e2804f0e56 2024-09-22T00:46:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:46:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:46:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:46:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-22T00:46:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd3 2024-09-22T00:46:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:46:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-22T00:46:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:46:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:46:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:46:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 5ad80959-7672-409c-8280-39fb185b4288 2024-09-22T00:46:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:46:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:46:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:46:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:46:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:46:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:46:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary4 2024-09-22T00:46:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-22T00:46:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:46:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary4 2024-09-22T00:46:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:46:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-22T00:46:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:46:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:46:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:46:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:46:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:46:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary4 2024-09-22T00:46:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:46:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-22T00:46:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary5 2024-09-22T00:46:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary4 2024-09-22T00:46:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-22T00:46:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-22T00:46:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:46:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:46:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-22T00:46:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:46:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:46:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:46:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:46:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-22T00:46:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:46:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:46:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-22T00:46:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:46:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-22T00:46:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:46:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-22T00:46:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:46:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2664 2024-09-22T00:46:28.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:28.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:46:28.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:28.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:28.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:46:28.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:46:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-22T00:46:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary5': 2024-09-22T00:46:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:46:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:46:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:46:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 34988b9b9df2 2024-09-22T00:46:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.34988b9b9df2 2024-09-22T00:46:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:46:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:46:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:46:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:46:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:46:28 2024 2024-09-22T00:46:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:46:28 2024 2024-09-22T00:46:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:46:28 2024 2024-09-22T00:46:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:46:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:46:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: e994f3be-a939-4371-b3d8-651240a01b3d 2024-09-22T00:46:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:46:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:46:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:46:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-22T00:46:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd4 2024-09-22T00:46:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:46:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-22T00:46:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:46:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:46:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:46:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 26bd9f3e-e583-4473-a0de-6876272b315b 2024-09-22T00:46:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:46:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:46:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:46:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:46:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:46:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:46:29.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary5 2024-09-22T00:46:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-22T00:46:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:46:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary5 2024-09-22T00:46:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:46:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-22T00:46:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:46:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:46:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:46:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:46:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:46:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:46:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary5 2024-09-22T00:46:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-22T00:46:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary6 2024-09-22T00:46:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-22T00:46:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:46:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:46:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary5 2024-09-22T00:46:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-22T00:46:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-22T00:46:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:46:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:46:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:46:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:46:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-22T00:46:29.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:46:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:46:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-22T00:46:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:46:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-22T00:46:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:46:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-22T00:46:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:46:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2666 2024-09-22T00:46:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-22T00:46:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary6': 2024-09-22T00:46:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:46:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:46:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:46:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 34aae6c33841 2024-09-22T00:46:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.34aae6c33841 2024-09-22T00:46:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:46:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:46:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:46:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:46:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:46:29 2024 2024-09-22T00:46:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:46:29 2024 2024-09-22T00:46:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:46:29 2024 2024-09-22T00:46:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:46:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:46:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 6334f7b1-bf34-4d11-af93-7a63cb657782 2024-09-22T00:46:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:46:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:46:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:46:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-22T00:46:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:46:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:46:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:46:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd5 2024-09-22T00:46:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:46:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-22T00:46:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:46:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:46:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:46:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: c88bd59c-1e0c-410e-81e4-d74e1f0351cd 2024-09-22T00:46:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:46:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:46:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:46:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:46:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:46:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:46:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary6 2024-09-22T00:46:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-22T00:46:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:46:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary6 2024-09-22T00:46:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:46:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-22T00:46:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:46:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:46:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:46:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:46:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:46:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:46:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary6 2024-09-22T00:46:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-22T00:46:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary7 2024-09-22T00:46:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-22T00:46:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary6 2024-09-22T00:46:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-22T00:46:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:46:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:46:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-22T00:46:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:46:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:46:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:46:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:46:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-22T00:46:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:46:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:46:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-22T00:46:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:46:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-22T00:46:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:46:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-22T00:46:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:46:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-22T00:46:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:46:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:46:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:46:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2670 2024-09-22T00:46:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary7': 2024-09-22T00:46:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:46:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:46:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:46:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 34bcbb6cbd20 2024-09-22T00:46:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.34bcbb6cbd20 2024-09-22T00:46:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:46:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:46:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:46:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:46:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:46:30 2024 2024-09-22T00:46:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:46:30 2024 2024-09-22T00:46:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:46:30 2024 2024-09-22T00:46:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:46:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:46:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 236d5724-41f0-4963-b51e-ce5af53eca01 2024-09-22T00:46:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:46:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:46:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:46:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-22T00:46:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd6 2024-09-22T00:46:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:46:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-22T00:46:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:46:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:46:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:46:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 6c2db6d3-b4c8-4f60-9db4-90ac009d3722 2024-09-22T00:46:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:46:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:46:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:46:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:46:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:46:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:46:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary7 2024-09-22T00:46:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-22T00:46:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:46:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary7 2024-09-22T00:46:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:46:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-22T00:46:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary7 2024-09-22T00:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-22T00:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary8 2024-09-22T00:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-22T00:46:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:46:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:46:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:46:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary7 2024-09-22T00:46:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-22T00:46:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-22T00:46:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:46:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:46:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:46:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:46:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-22T00:46:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:46:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:46:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-22T00:46:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:46:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-22T00:46:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:46:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-22T00:46:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2665 2024-09-22T00:46:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2668 2024-09-22T00:46:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:46:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:46:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:46:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:46:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2672 2024-09-22T00:46:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-22T00:46:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary8': 2024-09-22T00:46:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:46:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:46:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:46:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 34cd5671db1f 2024-09-22T00:46:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.34cd5671db1f 2024-09-22T00:46:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:46:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:46:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:46:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:46:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:46:31 2024 2024-09-22T00:46:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:46:31 2024 2024-09-22T00:46:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:46:31 2024 2024-09-22T00:46:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:46:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:46:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 054e263e-3a54-42f0-bae1-57f67ead0eff 2024-09-22T00:46:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:46:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:46:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:46:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-22T00:46:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd7 2024-09-22T00:46:32.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:46:32.053 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-22T00:46:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:46:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:46:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:46:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 195340ee-5d49-4853-9fd8-c395a02a04fa 2024-09-22T00:46:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:46:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:46:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:46:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:46:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:46:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:46:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary8 2024-09-22T00:46:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-22T00:46:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:46:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary8 2024-09-22T00:46:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:46:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-22T00:46:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:46:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:46:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:46:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:46:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:46:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary8 2024-09-22T00:46:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:46:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-22T00:46:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary8 2024-09-22T00:46:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary9 2024-09-22T00:46:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-22T00:46:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-22T00:46:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:46:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:46:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-22T00:46:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:46:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:46:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:46:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:46:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-22T00:46:32.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:46:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:46:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-22T00:46:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:46:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-22T00:46:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:46:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-22T00:46:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:46:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:46:32.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:46:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2671 2024-09-22T00:46:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:46:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-22T00:46:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2675 2024-09-22T00:46:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary9': 2024-09-22T00:46:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:46:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:46:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:46:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 34dd6dd97428 2024-09-22T00:46:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.34dd6dd97428 2024-09-22T00:46:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:46:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:46:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:46:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:46:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:46:32 2024 2024-09-22T00:46:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:46:32 2024 2024-09-22T00:46:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:46:32 2024 2024-09-22T00:46:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:46:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:46:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 7485789d-5063-4337-b05d-6cd26232c653 2024-09-22T00:46:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:46:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:46:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:46:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-22T00:46:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd8 2024-09-22T00:46:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:46:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-22T00:46:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:46:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:46:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:46:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 8d687bb6-c978-4515-b811-7010bd869cf2 2024-09-22T00:46:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:46:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:46:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:46:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:46:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:46:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:46:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary9 2024-09-22T00:46:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-22T00:46:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:46:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary9 2024-09-22T00:46:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:46:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-22T00:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-22T00:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary10 2024-09-22T00:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary9 2024-09-22T00:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-22T00:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:46:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-22T00:46:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:46:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary9 2024-09-22T00:46:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:46:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:46:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:46:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:46:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-22T00:46:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-22T00:46:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:46:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:46:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:46:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:46:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:46:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:46:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-22T00:46:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:46:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:46:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-22T00:46:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:46:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-22T00:46:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2674 2024-09-22T00:46:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2678 2024-09-22T00:46:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:46:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:46:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:46:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2677 2024-09-22T00:46:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:46:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-22T00:46:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary10': 2024-09-22T00:46:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:46:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:46:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:46:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 34ec445be57f 2024-09-22T00:46:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.34ec445be57f 2024-09-22T00:46:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:46:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:46:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:46:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:46:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:46:33 2024 2024-09-22T00:46:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:46:33 2024 2024-09-22T00:46:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:46:33 2024 2024-09-22T00:46:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:46:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:46:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: af2421d2-f6c8-49a0-8e4a-53d7470b473c 2024-09-22T00:46:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:46:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:46:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:46:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-22T00:46:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:46:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:46:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:46:34.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:34.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:46:34.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:34.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:34.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:46:34.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:46:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd9 2024-09-22T00:46:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:46:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-22T00:46:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:46:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:46:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:46:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: d14ac96c-6cf6-417e-b364-e562d65b32b9 2024-09-22T00:46:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:46:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:46:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:46:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:46:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:46:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:46:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:46:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:46:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:46:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:46:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary10 2024-09-22T00:46:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-22T00:46:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:46:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary10 2024-09-22T00:46:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:46:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-22T00:46:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:46:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:46:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 120478 2024-09-22T00:46:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary10 2024-09-22T00:46:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:46:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary10 2024-09-22T00:46:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-22T00:46:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:46:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:46:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:46:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:46:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-22T00:46:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2680 2024-09-22T00:46:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2684 2024-09-22T00:46:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:35.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:35.830 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:46:35.826+0000 7fab5fe19640 -1 reset not still connected to 0x562432c5d790 2024-09-22T00:46:35.830 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:46:35.826+0000 7fab5fe19640 -1 reset not still connected to 0x56243b0bba00 2024-09-22T00:46:35.831 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:46:35.830+0000 7fab5fe19640 -1 reset not still connected to 0x562448365e10 2024-09-22T00:46:35.846 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:46:35.842+0000 7fab5fe19640 -1 reset not still connected to 0x56240f368750 2024-09-22T00:46:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2682 2024-09-22T00:46:35.856 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:46:35.850+0000 7fab5fe19640 -1 reset not still connected to 0x56242a201ad0 2024-09-22T00:46:35.857 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:46:35.850+0000 7fab5fe19640 -1 reset not still connected to 0x56242a6e5450 2024-09-22T00:46:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:46:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:46:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:46:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:46:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:46:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:46:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2681 2024-09-22T00:46:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2683 2024-09-22T00:46:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:46:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:46:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:46:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:46:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:46:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:46:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:46:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:46:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:46:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2686 2024-09-22T00:46:37.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2685 2024-09-22T00:46:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2688 2024-09-22T00:46:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:46:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:46:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:46:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:46:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:46:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:46:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:46:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:46:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:46:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2689 2024-09-22T00:46:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:46:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:46:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:46:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:46:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:46:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:46:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2690 2024-09-22T00:46:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2687 2024-09-22T00:46:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2691 2024-09-22T00:46:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2692 2024-09-22T00:46:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:46:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:46:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:46:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:46:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:46:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:46:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:46:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:46:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:46:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2694 2024-09-22T00:46:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2693 2024-09-22T00:46:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:46:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:46:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:46:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2696 2024-09-22T00:46:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2695 2024-09-22T00:46:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:42.972 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:46:42.972+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a36835cd00 2024-09-22T00:46:42.975 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:46:42.972+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a3685708f0 2024-09-22T00:46:42.979 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:46:42.976+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a37a9c9930 2024-09-22T00:46:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:46:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:46:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:46:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:46:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:46:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:46:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:46:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:46:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:46:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:46:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:46:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:46:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2697 2024-09-22T00:46:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2698 2024-09-22T00:46:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2699 2024-09-22T00:46:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:46:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:46:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:46:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:46:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:46:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:46:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2702 2024-09-22T00:46:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2700 2024-09-22T00:46:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2701 2024-09-22T00:46:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:45.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:46:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:46:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:46:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:46:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:46:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:46:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2703 2024-09-22T00:46:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2704 2024-09-22T00:46:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:46:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:46:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:46:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2705 2024-09-22T00:46:47.641 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:46:47.638+0000 7f54f2d95640 -1 reset not still connected to 0x561804f6e000 2024-09-22T00:46:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:46:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:46:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:46:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:46:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:46:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:46:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2707 2024-09-22T00:46:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2706 2024-09-22T00:46:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:46:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:46:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:46:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:46:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:46:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:46:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:46:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:46:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:46:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2709 2024-09-22T00:46:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2708 2024-09-22T00:46:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:46:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:46:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:46:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:46:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:46:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:46:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2712 2024-09-22T00:46:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2711 2024-09-22T00:46:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:46:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:46:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:46:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2710 2024-09-22T00:46:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2713 2024-09-22T00:46:51.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2714 2024-09-22T00:46:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:46:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:46:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:46:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:46:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:46:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:46:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2715 2024-09-22T00:46:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:46:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:46:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:46:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:46:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:46:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:46:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2717 2024-09-22T00:46:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2716 2024-09-22T00:46:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:46:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:46:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:46:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:46:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:46:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:46:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:46:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:46:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:46:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2718 2024-09-22T00:46:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2719 2024-09-22T00:46:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:46:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:46:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:46:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:46:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:46:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:46:54.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2721 2024-09-22T00:46:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2720 2024-09-22T00:46:55.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:55.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:46:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:46:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:46:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2722 2024-09-22T00:46:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2724 2024-09-22T00:46:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2723 2024-09-22T00:46:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:46:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:46:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:46:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:46:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:46:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:46:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2725 2024-09-22T00:46:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:46:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:46:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:46:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:46:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:46:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:46:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2727 2024-09-22T00:46:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:46:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:46:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:46:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2726 2024-09-22T00:46:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:46:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:46:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:46:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:46:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:46:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:46:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:59.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2729 2024-09-22T00:46:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2728 2024-09-22T00:46:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:46:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:46:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:46:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:46:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:46:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:46:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:46:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:46:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:46:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:47:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2730 2024-09-22T00:47:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2732 2024-09-22T00:47:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:47:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:47:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:47:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2731 2024-09-22T00:47:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2734 2024-09-22T00:47:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2733 2024-09-22T00:47:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:47:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:47:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:47:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:47:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:47:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:47:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2735 2024-09-22T00:47:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:47:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:47:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:47:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:47:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:47:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:47:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:03.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2736 2024-09-22T00:47:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:47:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:47:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:47:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:47:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:47:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:47:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:47:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:47:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:47:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2737 2024-09-22T00:47:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2738 2024-09-22T00:47:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2739 2024-09-22T00:47:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:47:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:47:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:47:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:47:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:47:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:47:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2741 2024-09-22T00:47:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2740 2024-09-22T00:47:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2742 2024-09-22T00:47:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:47:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:47:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:47:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:05.837 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:47:05.836+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a3630960d0 2024-09-22T00:47:05.840 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:47:05.836+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a36835cd00 2024-09-22T00:47:05.840 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:47:05.836+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a3685708f0 2024-09-22T00:47:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2744 2024-09-22T00:47:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2743 2024-09-22T00:47:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:47:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:47:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:47:06.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:47:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:47:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:47:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:47:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:47:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:47:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:47:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:47:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:47:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2745 2024-09-22T00:47:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2746 2024-09-22T00:47:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2749 2024-09-22T00:47:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:47:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:47:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:47:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:47:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:47:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:47:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:47:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:47:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:47:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2748 2024-09-22T00:47:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2747 2024-09-22T00:47:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:47:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:47:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:47:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:47:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:47:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:47:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2751 2024-09-22T00:47:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2752 2024-09-22T00:47:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2750 2024-09-22T00:47:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:47:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:47:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:47:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:11.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2754 2024-09-22T00:47:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2753 2024-09-22T00:47:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:47:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:47:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:47:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:47:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:47:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:47:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:47:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:47:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:47:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:47:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:47:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:47:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2755 2024-09-22T00:47:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2757 2024-09-22T00:47:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2756 2024-09-22T00:47:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:47:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:47:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:47:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:47:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:47:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:47:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:47:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:47:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:47:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2758 2024-09-22T00:47:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2759 2024-09-22T00:47:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:47:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:47:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:47:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:47:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:47:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:47:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2762 2024-09-22T00:47:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2760 2024-09-22T00:47:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2761 2024-09-22T00:47:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:47:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:47:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:47:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2764 2024-09-22T00:47:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:47:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:47:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:47:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:47:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:47:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:47:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:47:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:47:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:47:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:47:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:47:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:47:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2763 2024-09-22T00:47:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2765 2024-09-22T00:47:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2766 2024-09-22T00:47:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:47:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:47:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:47:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2767 2024-09-22T00:47:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:47:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:47:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:47:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:47:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:47:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:47:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2769 2024-09-22T00:47:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2768 2024-09-22T00:47:19.687 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:19.686+0000 7fab5fe19640 -1 reset not still connected to 0x56243b0bba00 2024-09-22T00:47:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:19.693 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:19.690+0000 7fab5fe19640 -1 reset not still connected to 0x562448365e10 2024-09-22T00:47:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:47:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:47:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:47:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2772 2024-09-22T00:47:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2770 2024-09-22T00:47:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2771 2024-09-22T00:47:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:47:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:47:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:47:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:47:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:47:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:47:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2773 2024-09-22T00:47:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2774 2024-09-22T00:47:21.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:47:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:47:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:47:21.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:21.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:47:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:47:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:47:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2775 2024-09-22T00:47:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:47:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:47:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:47:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:47:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:47:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:47:22.845 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:22.842+0000 7fab5fe19640 -1 reset not still connected to 0x56242a201ad0 2024-09-22T00:47:22.846 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:22.842+0000 7fab5fe19640 -1 reset not still connected to 0x56242a6e5450 2024-09-22T00:47:23.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2776 2024-09-22T00:47:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2777 2024-09-22T00:47:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:47:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:47:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:47:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:47:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:47:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:47:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:47:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:47:23.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:47:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2778 2024-09-22T00:47:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2779 2024-09-22T00:47:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:47:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:47:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:47:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:47:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:47:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:47:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2781 2024-09-22T00:47:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2782 2024-09-22T00:47:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:47:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:47:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:47:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2780 2024-09-22T00:47:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:25.792 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:25.790+0000 7fab5fe19640 -1 reset not still connected to 0x5623fbedba00 2024-09-22T00:47:25.792 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:25.790+0000 7fab5fe19640 -1 reset not still connected to 0x56240755f1e0 2024-09-22T00:47:25.792 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:25.790+0000 7fab5fe19640 -1 reset not still connected to 0x56240f32b930 2024-09-22T00:47:25.792 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:25.790+0000 7fab5fe19640 -1 reset not still connected to 0x56240f368750 2024-09-22T00:47:25.792 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:25.790+0000 7fab5fe19640 -1 reset not still connected to 0x562424f42000 2024-09-22T00:47:25.792 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:25.790+0000 7fab5fe19640 -1 reset not still connected to 0x56242a201ad0 2024-09-22T00:47:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2783 2024-09-22T00:47:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2784 2024-09-22T00:47:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:47:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:47:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:47:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:47:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:47:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:47:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2785 2024-09-22T00:47:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:47:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:47:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:47:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:47:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:47:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:47:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2787 2024-09-22T00:47:28.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2786 2024-09-22T00:47:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:47:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:47:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:47:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:47:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:47:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:47:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:47:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:47:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:47:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2789 2024-09-22T00:47:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2788 2024-09-22T00:47:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:47:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:47:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:47:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:47:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:47:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:47:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2791 2024-09-22T00:47:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2792 2024-09-22T00:47:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:47:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:47:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:47:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2790 2024-09-22T00:47:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2794 2024-09-22T00:47:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2793 2024-09-22T00:47:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:47:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:47:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:47:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:31.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:31.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:47:31.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:47:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:47:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:47:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:47:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:47:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:47:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:47:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:47:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2795 2024-09-22T00:47:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:32.759 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:32.754+0000 7fab5fe19640 -1 reset not still connected to 0x56240f32b930 2024-09-22T00:47:32.759 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:32.754+0000 7fab5fe19640 -1 reset not still connected to 0x56240f368750 2024-09-22T00:47:32.759 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:32.754+0000 7fab5fe19640 -1 reset not still connected to 0x562424f42000 2024-09-22T00:47:32.759 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:32.754+0000 7fab5fe19640 -1 reset not still connected to 0x56242a201ad0 2024-09-22T00:47:32.759 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:32.754+0000 7fab5fe19640 -1 reset not still connected to 0x56242a6e5450 2024-09-22T00:47:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2796 2024-09-22T00:47:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2797 2024-09-22T00:47:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:47:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:47:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:47:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:47:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:47:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:47:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:47:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:47:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:47:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2798 2024-09-22T00:47:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2799 2024-09-22T00:47:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:47:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:47:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:47:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:47:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:47:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:47:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2802 2024-09-22T00:47:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2801 2024-09-22T00:47:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2800 2024-09-22T00:47:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:47:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:47:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:47:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2803 2024-09-22T00:47:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2804 2024-09-22T00:47:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:47:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:47:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:47:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:47:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:47:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:47:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2805 2024-09-22T00:47:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:47:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:47:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:47:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:47:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:47:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:47:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2806 2024-09-22T00:47:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:47:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:47:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:47:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:47:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:47:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:47:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:47:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:47:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:47:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2808 2024-09-22T00:47:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2809 2024-09-22T00:47:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2807 2024-09-22T00:47:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:39.702 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:39.698+0000 7fab5fe19640 -1 reset not still connected to 0x56240f32b930 2024-09-22T00:47:39.702 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:39.698+0000 7fab5fe19640 -1 reset not still connected to 0x56240f368750 2024-09-22T00:47:39.702 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:39.698+0000 7fab5fe19640 -1 reset not still connected to 0x56242a201ad0 2024-09-22T00:47:39.702 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:47:39.698+0000 7fab5fe19640 -1 reset not still connected to 0x56242a6e5450 2024-09-22T00:47:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:47:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:47:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:47:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:47:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:47:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:47:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2810 2024-09-22T00:47:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2812 2024-09-22T00:47:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2811 2024-09-22T00:47:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:47:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:47:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:47:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2813 2024-09-22T00:47:41.744 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:47:41.739+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a3685708f0 2024-09-22T00:47:41.744 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:47:41.739+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a37a9c9930 2024-09-22T00:47:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2814 2024-09-22T00:47:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:47:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:47:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:47:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:47:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:47:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:47:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:47:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:47:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:47:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:47:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:47:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:47:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2815 2024-09-22T00:47:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2816 2024-09-22T00:47:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:47:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:47:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:47:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:47:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:47:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:47:43.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:43.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:47:43.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:47:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:47:43.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2819 2024-09-22T00:47:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2818 2024-09-22T00:47:44.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2817 2024-09-22T00:47:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:47:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:47:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:47:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:47:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:47:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:47:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2822 2024-09-22T00:47:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2820 2024-09-22T00:47:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:47:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:47:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:47:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2821 2024-09-22T00:47:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2823 2024-09-22T00:47:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2824 2024-09-22T00:47:46.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:46.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:47:46.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:46.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:47:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:47:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:47:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:47:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:47:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:47:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:47:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:47:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:47:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:47:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:47:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2825 2024-09-22T00:47:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2827 2024-09-22T00:47:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2826 2024-09-22T00:47:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:47:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:47:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:47:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:47:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:47:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:47:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:47:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:47:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:47:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2829 2024-09-22T00:47:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2828 2024-09-22T00:47:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:47:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:47:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:47:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:47:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:47:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:47:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2830 2024-09-22T00:47:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2832 2024-09-22T00:47:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2831 2024-09-22T00:47:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:47:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:47:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:47:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2834 2024-09-22T00:47:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2833 2024-09-22T00:47:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:47:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:47:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:47:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:47:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:47:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:47:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:47:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:47:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:47:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:47:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:47:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:47:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2835 2024-09-22T00:47:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:47:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:47:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:47:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:47:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:47:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:47:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:47:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:47:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:47:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2837 2024-09-22T00:47:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2836 2024-09-22T00:47:53.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2838 2024-09-22T00:47:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2839 2024-09-22T00:47:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:54.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:54.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:47:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:47:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:47:54.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:47:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:47:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:47:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2840 2024-09-22T00:47:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2841 2024-09-22T00:47:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2842 2024-09-22T00:47:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:55.733 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:47:55.727+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a36835cd00 2024-09-22T00:47:55.733 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:47:55.727+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a3685708f0 2024-09-22T00:47:55.733 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:47:55.731+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a3685708f0 2024-09-22T00:47:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:47:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:47:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:47:56.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2843 2024-09-22T00:47:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2844 2024-09-22T00:47:56.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:47:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:47:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:47:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:47:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:47:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:47:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2845 2024-09-22T00:47:57.678 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:47:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:47:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:47:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:47:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:47:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:47:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2847 2024-09-22T00:47:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2846 2024-09-22T00:47:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:47:58.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:58.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:58.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:47:58.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:47:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:47:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:47:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:47:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:47:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:47:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:47:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2849 2024-09-22T00:47:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2848 2024-09-22T00:47:59.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:47:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:47:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:47:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:47:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:47:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:47:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:47:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:47:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:47:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:48:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2850 2024-09-22T00:48:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2852 2024-09-22T00:48:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2851 2024-09-22T00:48:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:48:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:48:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:48:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2854 2024-09-22T00:48:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2853 2024-09-22T00:48:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:48:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:48:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:48:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:48:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:48:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:48:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2855 2024-09-22T00:48:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:48:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:48:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:48:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:48:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:48:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:48:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2857 2024-09-22T00:48:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2856 2024-09-22T00:48:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:48:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:48:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:48:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:48:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:03.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:03.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:48:03.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:48:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:48:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:48:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:48:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2858 2024-09-22T00:48:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2859 2024-09-22T00:48:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:48:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:48:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:48:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:48:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:48:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:48:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2861 2024-09-22T00:48:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2862 2024-09-22T00:48:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2860 2024-09-22T00:48:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:48:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:48:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:48:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2864 2024-09-22T00:48:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2863 2024-09-22T00:48:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:48:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:48:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:48:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:48:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:48:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:48:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2865 2024-09-22T00:48:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:48:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:48:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:48:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:48:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:48:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:48:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2867 2024-09-22T00:48:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2866 2024-09-22T00:48:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:48:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:48:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:48:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:48:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:48:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:48:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:48:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:48:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:48:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2868 2024-09-22T00:48:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:48:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:48:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:48:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:48:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:48:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:48:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2869 2024-09-22T00:48:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2871 2024-09-22T00:48:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2870 2024-09-22T00:48:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2872 2024-09-22T00:48:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:48:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:48:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:48:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2873 2024-09-22T00:48:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2874 2024-09-22T00:48:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:48:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:48:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:48:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:48:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:48:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:48:12.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2875 2024-09-22T00:48:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:48:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:48:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:48:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:48:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:48:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:48:13.584 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:48:13.579+0000 7f84953c5640 -1 reset not still connected to 0x56370e62d2b0 2024-09-22T00:48:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2877 2024-09-22T00:48:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2876 2024-09-22T00:48:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:48:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:48:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:48:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:48:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:48:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:48:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:48:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:48:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:48:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2879 2024-09-22T00:48:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2878 2024-09-22T00:48:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:48:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:48:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:48:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:48:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:48:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:48:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2882 2024-09-22T00:48:15.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2880 2024-09-22T00:48:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2881 2024-09-22T00:48:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:48:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:48:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:48:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2884 2024-09-22T00:48:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2883 2024-09-22T00:48:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:48:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:48:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:48:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:48:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:48:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:48:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:48:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:48:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:48:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:48:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:48:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:48:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2885 2024-09-22T00:48:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2887 2024-09-22T00:48:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2886 2024-09-22T00:48:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:48:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:48:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:48:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:48:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:48:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:48:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:18.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:48:18.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:18.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:18.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:48:18.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:48:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:18.917 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:48:18.913+0000 7fab5fe19640 -1 reset not still connected to 0x56240f368750 2024-09-22T00:48:18.921 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:48:18.917+0000 7fab5fe19640 -1 reset not still connected to 0x562424f42000 2024-09-22T00:48:18.921 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:48:18.917+0000 7fab5fe19640 -1 reset not still connected to 0x56242a201ad0 2024-09-22T00:48:18.922 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:48:18.917+0000 7fab5fe19640 -1 reset not still connected to 0x56242a6e5450 2024-09-22T00:48:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2888 2024-09-22T00:48:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2889 2024-09-22T00:48:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:48:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:48:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:48:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:48:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:48:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:48:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2891 2024-09-22T00:48:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2892 2024-09-22T00:48:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2890 2024-09-22T00:48:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:48:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:48:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:48:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2893 2024-09-22T00:48:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2894 2024-09-22T00:48:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:48:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:48:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:48:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:48:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:48:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:48:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:48:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:48:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:48:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:48:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:48:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:48:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2895 2024-09-22T00:48:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2897 2024-09-22T00:48:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2896 2024-09-22T00:48:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:48:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:48:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:48:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:48:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:48:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:48:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:48:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2899 2024-09-22T00:48:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2898 2024-09-22T00:48:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:48:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:48:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:48:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:48:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:48:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:48:25.632 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2901 2024-09-22T00:48:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2900 2024-09-22T00:48:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2902 2024-09-22T00:48:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:48:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:48:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:48:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2904 2024-09-22T00:48:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2903 2024-09-22T00:48:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:48:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:48:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:48:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:48:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:48:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:48:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2905 2024-09-22T00:48:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:48:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:48:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:48:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:48:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:48:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:48:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2906 2024-09-22T00:48:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2907 2024-09-22T00:48:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:48:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:48:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:48:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:48:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:48:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:48:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:48:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:48:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:48:28.793 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:48:28.789+0000 7fab5fe19640 -1 reset not still connected to 0x5623fbedba00 2024-09-22T00:48:28.793 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:48:28.789+0000 7fab5fe19640 -1 reset not still connected to 0x56240755f1e0 2024-09-22T00:48:28.794 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:48:28.789+0000 7fab5fe19640 -1 reset not still connected to 0x56240f32b930 2024-09-22T00:48:28.794 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:48:28.789+0000 7fab5fe19640 -1 reset not still connected to 0x56240f368750 2024-09-22T00:48:28.794 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:48:28.789+0000 7fab5fe19640 -1 reset not still connected to 0x56242a201ad0 2024-09-22T00:48:28.795 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:48:28.793+0000 7fab5fe19640 -1 reset not still connected to 0x56242a6e5450 2024-09-22T00:48:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2908 2024-09-22T00:48:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2909 2024-09-22T00:48:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:48:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:48:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:48:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:48:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:48:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:48:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2910 2024-09-22T00:48:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2911 2024-09-22T00:48:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2912 2024-09-22T00:48:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:48:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:48:30.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:48:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2913 2024-09-22T00:48:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2914 2024-09-22T00:48:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:48:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:48:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:48:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:48:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:48:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:48:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2915 2024-09-22T00:48:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:48:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:48:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:48:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:48:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:48:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:48:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2916 2024-09-22T00:48:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:48:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:48:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:48:33.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:33.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:48:33.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:33.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:33.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:48:33.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:48:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2917 2024-09-22T00:48:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:48:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:48:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:48:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2918 2024-09-22T00:48:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2919 2024-09-22T00:48:34.686 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:48:34.679+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a36835cd00 2024-09-22T00:48:34.691 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:48:34.687+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a3685708f0 2024-09-22T00:48:34.699 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:48:34.695+0000 7fab1f496640 -1 reset not still connected to 0x55fa608a7110 2024-09-22T00:48:34.699 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:48:34.695+0000 7fab1f496640 -1 reset not still connected to 0x55fa6b391ee0 2024-09-22T00:48:34.699 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:48:34.695+0000 7fab1f496640 -1 reset not still connected to 0x55fa6d57d1e0 2024-09-22T00:48:34.699 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:48:34.695+0000 7fab1f496640 -1 reset not still connected to 0x55fa820d08f0 2024-09-22T00:48:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:48:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:48:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:48:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:48:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:48:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:48:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2922 2024-09-22T00:48:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2920 2024-09-22T00:48:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2921 2024-09-22T00:48:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:48:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:48:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:48:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2924 2024-09-22T00:48:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2923 2024-09-22T00:48:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:48:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:48:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:48:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:48:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:48:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:48:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:48:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:48:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:48:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2925 2024-09-22T00:48:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:48:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:48:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:48:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:48:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:48:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:48:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2927 2024-09-22T00:48:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:48:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:48:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:48:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2926 2024-09-22T00:48:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:48:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:48:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:48:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2928 2024-09-22T00:48:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2929 2024-09-22T00:48:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:48:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:48:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:48:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:48:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:48:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:48:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2930 2024-09-22T00:48:40.759 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:48:40.755+0000 7f84953c5640 -1 reset not still connected to 0x5636f85f0d00 2024-09-22T00:48:40.765 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:48:40.763+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b32ea0 2024-09-22T00:48:40.766 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:48:40.763+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b332b0 2024-09-22T00:48:40.766 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:48:40.763+0000 7f84953c5640 -1 reset not still connected to 0x5636f8ec80d0 2024-09-22T00:48:40.766 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:48:40.763+0000 7f84953c5640 -1 reset not still connected to 0x5636f9c289c0 2024-09-22T00:48:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2931 2024-09-22T00:48:40.769 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:48:40.767+0000 7f84953c5640 -1 reset not still connected to 0x56370e62d2b0 2024-09-22T00:48:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2932 2024-09-22T00:48:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:48:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:48:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:48:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:41.596 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:48:41.597+0000 7fab5fe19640 -1 reset not still connected to 0x56243b0bba00 2024-09-22T00:48:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2933 2024-09-22T00:48:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2934 2024-09-22T00:48:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:48:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:48:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:48:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:48:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:48:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:48:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:48:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:48:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:48:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:48:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:48:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:48:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2935 2024-09-22T00:48:42.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2936 2024-09-22T00:48:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2937 2024-09-22T00:48:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:48:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:48:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:48:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:48:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:48:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:48:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:48:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:48:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:48:44.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2938 2024-09-22T00:48:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2939 2024-09-22T00:48:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:48:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:48:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:48:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:48:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:48:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:48:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2941 2024-09-22T00:48:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2940 2024-09-22T00:48:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2942 2024-09-22T00:48:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:48:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:48:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:48:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2944 2024-09-22T00:48:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2943 2024-09-22T00:48:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:48:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:48:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:48:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:48:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:48:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:48:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:48:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:48:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:48:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:48:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:48:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:48:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2945 2024-09-22T00:48:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2947 2024-09-22T00:48:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:48:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:48:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:48:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2946 2024-09-22T00:48:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:48:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:48:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:48:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:48:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:48:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:48:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2949 2024-09-22T00:48:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2948 2024-09-22T00:48:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:48:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:48:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:48:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:48:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:48:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:48:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2952 2024-09-22T00:48:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2950 2024-09-22T00:48:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2951 2024-09-22T00:48:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:48:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:48:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:48:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2953 2024-09-22T00:48:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2954 2024-09-22T00:48:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:48:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:48:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:48:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:48:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:48:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:48:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:48:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:48:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:48:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:48:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:48:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:48:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2955 2024-09-22T00:48:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2956 2024-09-22T00:48:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2957 2024-09-22T00:48:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:48:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 120605 2024-09-22T00:48:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:48:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:48:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:48:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:48:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:48:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:48:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:53.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:54.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2958 2024-09-22T00:48:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2959 2024-09-22T00:48:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:48:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:48:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:48:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:48:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 120732 2024-09-22T00:48:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2960 2024-09-22T00:48:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2961 2024-09-22T00:48:55.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:48:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:48:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:48:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2962 2024-09-22T00:48:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:48:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 120859 2024-09-22T00:48:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:48:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 121006 2024-09-22T00:48:57.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2963 2024-09-22T00:48:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:48:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:48:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:48:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:48:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 121153 2024-09-22T00:48:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:48:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:48:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:48:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:48:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:48:58.687 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:48:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:48:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 121300 2024-09-22T00:48:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2964 2024-09-22T00:48:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:48:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:48:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 121449 2024-09-22T00:49:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2965 2024-09-22T00:49:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:49:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 121597 2024-09-22T00:49:00.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:49:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:49:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 121806 2024-09-22T00:49:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:49:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:49:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 120479 2024-09-22T00:51:01.110 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:51:01.107+0000 7fab5fe19640 -1 reset not still connected to 0x562448365e10 2024-09-22T00:51:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:51:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:51:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:51:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 120606 2024-09-22T00:51:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:51:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:51:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:51:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 120733 2024-09-22T00:51:27.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:51:27.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:51:27.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:51:27.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 120860 2024-09-22T00:51:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:51:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:51:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:51:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 121007 2024-09-22T00:51:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:51:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:51:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:51:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 121154 2024-09-22T00:51:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:51:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:51:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:51:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 121301 2024-09-22T00:51:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:51:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:51:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:51:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 121450 2024-09-22T00:51:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:51:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:51:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:51:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 121598 2024-09-22T00:51:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:51:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:51:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:51:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 121807 2024-09-22T00:51:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:51:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:51:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:51:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:51:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:51:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 1 2024-09-22T00:51:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:51:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd0 2024-09-22T00:51:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary1 2024-09-22T00:51:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary1 2024-09-22T00:51:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:51:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary1 2024-09-22T00:51:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:51:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:51:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 2 2024-09-22T00:51:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd1 2024-09-22T00:51:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary2 2024-09-22T00:51:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary2 2024-09-22T00:51:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:51:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:51:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary2 2024-09-22T00:51:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 3 2024-09-22T00:51:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:51:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd2 2024-09-22T00:51:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary3 2024-09-22T00:51:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary3 2024-09-22T00:51:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:51:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary3 2024-09-22T00:51:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:51:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:51:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 4 2024-09-22T00:51:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd3 2024-09-22T00:51:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary4 2024-09-22T00:51:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary4 2024-09-22T00:51:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 5 2024-09-22T00:51:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:51:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:51:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary4 2024-09-22T00:51:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:51:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd4 2024-09-22T00:51:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary5 2024-09-22T00:51:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary5 2024-09-22T00:51:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:51:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary5 2024-09-22T00:51:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:51:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 6 2024-09-22T00:51:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:51:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd5 2024-09-22T00:51:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary6 2024-09-22T00:51:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary6 2024-09-22T00:51:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:51:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary6 2024-09-22T00:51:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:51:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:51:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 7 2024-09-22T00:51:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:51:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 8 2024-09-22T00:51:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:51:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd7 2024-09-22T00:51:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd6 2024-09-22T00:51:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary8 2024-09-22T00:51:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary8 2024-09-22T00:51:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary7 2024-09-22T00:51:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:51:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary7 2024-09-22T00:51:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary8 2024-09-22T00:51:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:51:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary7 2024-09-22T00:51:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:51:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:51:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 9 2024-09-22T00:51:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd8 2024-09-22T00:51:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary9 2024-09-22T00:51:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary9 2024-09-22T00:51:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:51:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary9 2024-09-22T00:51:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:51:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 10 2024-09-22T00:51:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:51:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 127924 2024-09-22T00:51:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd9 2024-09-22T00:51:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary10 2024-09-22T00:51:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary10 2024-09-22T00:51:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:51:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary10 2024-09-22T00:51:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-22T00:51:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:51:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-22T00:51:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:51:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-22T00:51:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:51:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-22T00:51:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:51:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:51:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-22T00:51:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-22T00:51:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:51:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:51:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:51:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-22T00:51:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-22T00:51:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:51:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-22T00:51:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-22T00:51:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:51:49.602 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:51:49.598+0000 7fab5fe19640 -1 reset not still connected to 0x56240fbd5ad0 2024-09-22T00:51:49.602 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:51:49.598+0000 7fab5fe19640 -1 reset not still connected to 0x5624214afe10 2024-09-22T00:51:49.604 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:51:49.602+0000 7fab5fe19640 -1 reset not still connected to 0x56242a6e49c0 2024-09-22T00:51:49.604 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:51:49.602+0000 7fab5fe19640 -1 reset not still connected to 0x56242f21d1e0 2024-09-22T00:51:49.604 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:51:49.602+0000 7fab5fe19640 -1 reset not still connected to 0x562432c5d790 2024-09-22T00:51:49.604 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:51:49.602+0000 7fab5fe19640 -1 reset not still connected to 0x56243b0bba00 2024-09-22T00:51:49.604 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:51:49.602+0000 7fab5fe19640 -1 reset not still connected to 0x562448365e10 2024-09-22T00:52:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=e73fdbd3ca8fdcb6ffae955837fb3b32 2024-09-22T00:52:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-22T00:52:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-22T00:52:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:52:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-22T00:52:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=8a5e4feb46e0e33ce41483fc605ed340 2024-09-22T00:52:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-22T00:52:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=413e91c4573c1a50cb58099ed0d715d2 2024-09-22T00:52:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-22T00:52:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=0e31c7ceb06486d614f96c9eaf6bfbcc 2024-09-22T00:52:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-22T00:52:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-22T00:52:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:52:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-22T00:52:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-22T00:52:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:52:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-22T00:52:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-22T00:52:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:52:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-22T00:52:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=1c1ca4960d284b62d13ec703352027d8 2024-09-22T00:52:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-22T00:52:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=c6a623473fd18db36a355a5730937b9b 2024-09-22T00:52:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-22T00:52:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-22T00:52:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:52:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-22T00:52:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-22T00:52:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:52:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-22T00:52:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=d66ac50f2e527babdf05301cb551916e 2024-09-22T00:52:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-22T00:52:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=64a3bde46bdc6f043f62b7742096054d 2024-09-22T00:52:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-22T00:52:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=850a824ddf0dcd95a87afb327e4cb8b5 2024-09-22T00:52:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-22T00:52:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-22T00:52:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:52:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-22T00:52:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-22T00:52:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:52:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-22T00:52:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-22T00:52:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:52:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-22T00:52:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=6b2da8a3960069b45c3ae05ae74347ed 2024-09-22T00:52:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-22T00:52:09.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-22T00:52:09.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:09.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:09.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:52:09.599 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-22T00:52:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:52:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:52:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-22T00:52:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:52:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-22T00:52:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:52:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:10.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:52:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:52:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:52:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:52:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-22T00:52:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:52:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-22T00:52:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:52:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-22T00:52:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:52:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-22T00:52:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:52:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:52:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:52:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-22T00:52:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:52:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:52:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-22T00:52:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:52:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:52:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:52:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-22T00:52:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:52:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:52:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-22T00:52:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:52:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-22T00:52:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:52:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-22T00:52:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:52:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-22T00:52:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:52:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:52:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-22T00:52:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:52:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:52:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-22T00:52:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:52:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:52:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-22T00:52:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:52:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:52:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:52:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 054e263e-3a54-42f0-bae1-57f67ead0eff 2024-09-22T00:52:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10 2024-09-22T00:52:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:52:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10' 2024-09-22T00:52:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:52:11.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:11.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:11.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:52:11.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 7485789d-5063-4337-b05d-6cd26232c653 2024-09-22T00:52:11.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:11.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10 2024-09-22T00:52:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:52:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10' 2024-09-22T00:52:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:52:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 236d5724-41f0-4963-b51e-ce5af53eca01 2024-09-22T00:52:11.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:11.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:11.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:11.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10 2024-09-22T00:52:11.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:11.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:52:11.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:11.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:11.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10' 2024-09-22T00:52:11.518 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:11.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:11.519 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:52:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6334f7b1-bf34-4d11-af93-7a63cb657782 2024-09-22T00:52:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10 2024-09-22T00:52:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:52:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10' 2024-09-22T00:52:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:52:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:11.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:11.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:11.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:11.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:11.524 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:52:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:52:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-22 00:52:10' 2024-09-22T00:52:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:52:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:10' 2024-09-22T00:52:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-22 00:52:10' 2024-09-22T00:52:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:52:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:10' 2024-09-22T00:52:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:52:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: af2421d2-f6c8-49a0-8e4a-53d7470b473c 2024-09-22T00:52:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10 2024-09-22T00:52:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:52:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10' 2024-09-22T00:52:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-22 00:52:10' 2024-09-22T00:52:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:52:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:10' 2024-09-22T00:52:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:52:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-22 00:52:10' 2024-09-22T00:52:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:52:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:10' 2024-09-22T00:52:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-22 00:52:10' 2024-09-22T00:52:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:52:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:10' 2024-09-22T00:52:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:52:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8685aec3-047e-4aad-924b-f779ecacae9d 2024-09-22T00:52:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10 2024-09-22T00:52:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:52:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10' 2024-09-22T00:52:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:52:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-22 00:52:10' 2024-09-22T00:52:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:52:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:10' 2024-09-22T00:52:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4503552.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966129,"remote_snapshot_timestamp":1726966129,"replay_state":"idle"}2024-09-22 00:52:05image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":180224.0,"last_snapshot_bytes":122880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966132,"remote_snapshot_timestamp":1726966132,"replay_state":"idle"}2024-09-22 00:52:05image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1703936.0,"last_snapshot_bytes":143360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966132,"remote_snapshot_timestamp":1726966132,"replay_state":"idle"}2024-09-22 00:52:05image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1710080.0,"last_snapshot_bytes":1597440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966133,"remote_snapshot_timestamp":1726966133,"replay_state":"idle"}2024-09-22 00:52:05image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10' 2024-09-22T00:52:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4503552.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966129,"remote_snapshot_timestamp":1726966129,"replay_state":"idle"}2024-09-22 00:52:05image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":180224.0,"last_snapshot_bytes":122880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966132,"remote_snapshot_timestamp":1726966132,"replay_state":"idle"}2024-09-22 00:52:05image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1703936.0,"last_snapshot_bytes":143360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966132,"remote_snapshot_timestamp":1726966132,"replay_state":"idle"}2024-09-22 00:52:05image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1710080.0,"last_snapshot_bytes":1597440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966133,"remote_snapshot_timestamp":1726966133,"replay_state":"idle"}2024-09-22 00:52:05image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10' 2024-09-22T00:52:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:52:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4503552.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966129,"remote_snapshot_timestamp":1726966129,"replay_state":"idle"}2024-09-22 00:52:05image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":180224.0,"last_snapshot_bytes":122880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966132,"remote_snapshot_timestamp":1726966132,"replay_state":"idle"}2024-09-22 00:52:05image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1703936.0,"last_snapshot_bytes":143360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966132,"remote_snapshot_timestamp":1726966132,"replay_state":"idle"}2024-09-22 00:52:05image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1710080.0,"last_snapshot_bytes":1597440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966133,"remote_snapshot_timestamp":1726966133,"replay_state":"idle"}2024-09-22 00:52:05image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10' 2024-09-22T00:52:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:52:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:52:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4503552.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966129,"remote_snapshot_timestamp":1726966129,"replay_state":"idle"}2024-09-22 00:52:05image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":180224.0,"last_snapshot_bytes":122880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966132,"remote_snapshot_timestamp":1726966132,"replay_state":"idle"}2024-09-22 00:52:05image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1703936.0,"last_snapshot_bytes":143360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966132,"remote_snapshot_timestamp":1726966132,"replay_state":"idle"}2024-09-22 00:52:05image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1710080.0,"last_snapshot_bytes":1597440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966133,"remote_snapshot_timestamp":1726966133,"replay_state":"idle"}2024-09-22 00:52:05image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10' 2024-09-22T00:52:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:52:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4503552.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966129,"remote_snapshot_timestamp":1726966129,"replay_state":"idle"}2024-09-22 00:52:05image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":180224.0,"last_snapshot_bytes":122880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966132,"remote_snapshot_timestamp":1726966132,"replay_state":"idle"}2024-09-22 00:52:05image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1703936.0,"last_snapshot_bytes":143360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966132,"remote_snapshot_timestamp":1726966132,"replay_state":"idle"}2024-09-22 00:52:05image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1710080.0,"last_snapshot_bytes":1597440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966133,"remote_snapshot_timestamp":1726966133,"replay_state":"idle"}2024-09-22 00:52:05image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10' 2024-09-22T00:52:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:52:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4503552.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966129,"remote_snapshot_timestamp":1726966129,"replay_state":"idle"}2024-09-22 00:52:05image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stopping_replayremote image demoted2024-09-22 00:52:11image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1703936.0,"last_snapshot_bytes":143360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966132,"remote_snapshot_timestamp":1726966132,"replay_state":"idle"}2024-09-22 00:52:05image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stopping_replayremote image demoted2024-09-22 00:52:11image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10' 2024-09-22T00:52:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:52:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:52:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:52:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:52:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:52:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-22T00:52:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:52:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-22T00:52:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:52:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-22T00:52:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:52:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-22T00:52:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:52:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:52:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:52:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-22T00:52:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:52:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-22T00:52:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:52:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-22T00:52:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:52:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:52:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-22T00:52:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:52:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:52:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-22T00:52:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:52:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:52:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-22T00:52:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:52:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:52:12.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:52:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6a888d77-180b-47c1-9533-64e2804f0e56 2024-09-22T00:52:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10 2024-09-22T00:52:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:52:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:11' 2024-09-22T00:52:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:52:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-22 00:52:10' 2024-09-22T00:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:10' 2024-09-22T00:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:52:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 747e5399-1569-4c2b-9ca7-3e020ff9d82c 2024-09-22T00:52:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10 2024-09-22T00:52:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:52:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:11' 2024-09-22T00:52:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:52:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-22 00:52:10' 2024-09-22T00:52:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:52:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:10' 2024-09-22T00:52:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:52:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 951a197c-f95a-4cd5-b5ed-190987539508 2024-09-22T00:52:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10 2024-09-22T00:52:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:52:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:11' 2024-09-22T00:52:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:52:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-22 00:52:10' 2024-09-22T00:52:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:52:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:10' 2024-09-22T00:52:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:12.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:52:12.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e994f3be-a939-4371-b3d8-651240a01b3d 2024-09-22T00:52:12.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:12.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:12.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:12.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10 2024-09-22T00:52:12.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:12.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:52:12.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:12.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:11' 2024-09-22T00:52:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:52:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-22 00:52:10' 2024-09-22T00:52:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:52:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:10' 2024-09-22T00:52:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:11image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:11image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:11image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:11image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10' 2024-09-22T00:52:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:52:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:52:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-22T00:52:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:52:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:12.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:12.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:12.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:11image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:11image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:11image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:11image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10' 2024-09-22T00:52:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:52:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:11image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:11image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:11image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:11image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10' 2024-09-22T00:52:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:52:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:52:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:52:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-22T00:52:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:52:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-22T00:52:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:52:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:11image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:11image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:11image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:11image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:52:10cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:52:10' 2024-09-22T00:52:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:52:12.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:12.663 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:52:12.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:12.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:12.666 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:12.667 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-22T00:52:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:52:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-22T00:52:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:52:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:12.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:12.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-22T00:52:12.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-22T00:52:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:52:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-22T00:52:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-22T00:52:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:52:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-22T00:52:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:52:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-22T00:52:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-22T00:52:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-22T00:52:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:52:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-22T00:52:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-22T00:52:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:52:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-22T00:52:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:52:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8685aec3-047e-4aad-924b-f779ecacae9d 2024-09-22T00:52:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10 2024-09-22T00:52:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:52:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10' 2024-09-22T00:52:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:52:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 054e263e-3a54-42f0-bae1-57f67ead0eff 2024-09-22T00:52:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10 2024-09-22T00:52:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:52:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10' 2024-09-22T00:52:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:52:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:52:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:12.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:12.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:12.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-22 00:52:10' 2024-09-22T00:52:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:52:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:10' 2024-09-22T00:52:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-22 00:52:10' 2024-09-22T00:52:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:52:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:10' 2024-09-22T00:52:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:52:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6334f7b1-bf34-4d11-af93-7a63cb657782 2024-09-22T00:52:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10 2024-09-22T00:52:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:52:12.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:12.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:12.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10' 2024-09-22T00:52:12.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:52:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:52:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 236d5724-41f0-4963-b51e-ce5af53eca01 2024-09-22T00:52:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10 2024-09-22T00:52:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:52:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10' 2024-09-22T00:52:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:52:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-22 00:52:10' 2024-09-22T00:52:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:52:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:10' 2024-09-22T00:52:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-22 00:52:10' 2024-09-22T00:52:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:52:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:10' 2024-09-22T00:52:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:52:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 7485789d-5063-4337-b05d-6cd26232c653 2024-09-22T00:52:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10 2024-09-22T00:52:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:52:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10' 2024-09-22T00:52:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:52:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: af2421d2-f6c8-49a0-8e4a-53d7470b473c 2024-09-22T00:52:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10 2024-09-22T00:52:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:52:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10' 2024-09-22T00:52:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:52:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:52:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:13.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:13.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:13.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:13.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-22 00:52:10' 2024-09-22T00:52:13.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:13.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:13.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:52:13.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:13.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:10' 2024-09-22T00:52:13.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:13.010 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:13.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:13.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:13.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:13.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:13.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:13.013 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-22 00:52:10' 2024-09-22T00:52:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:52:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:10' 2024-09-22T00:52:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:13.167 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:52:13.161+0000 7fab1f496640 -1 reset not still connected to 0x55fa87241520 2024-09-22T00:52:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10' 2024-09-22T00:52:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:52:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:52:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-22T00:52:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:52:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:52:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-22T00:52:13.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10' 2024-09-22T00:52:13.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:13.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:52:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:52:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-22T00:52:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:52:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:52:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-22T00:52:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10' 2024-09-22T00:52:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:52:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:52:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10' 2024-09-22T00:52:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:52:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-22T00:52:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:52:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:52:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-22T00:52:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:52:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10' 2024-09-22T00:52:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:52:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-22T00:52:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:52:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:52:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-22T00:52:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:52:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-22T00:52:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:52:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:52:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-22T00:52:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10' 2024-09-22T00:52:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:52:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:10' 2024-09-22T00:52:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:52:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:13.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-22T00:52:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:52:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:52:13.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-22T00:52:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-22T00:52:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:52:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-22T00:52:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:52:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6a888d77-180b-47c1-9533-64e2804f0e56 2024-09-22T00:52:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:11 2024-09-22T00:52:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:52:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10' 2024-09-22T00:52:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:52:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-22T00:52:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:52:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:13.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:13.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-22T00:52:13.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:13.658 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-22T00:52:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:52:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-22T00:52:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:11' 2024-09-22T00:52:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-22 00:52:11' 2024-09-22T00:52:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:52:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:11' 2024-09-22T00:52:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-22T00:52:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:52:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-22T00:52:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:52:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 951a197c-f95a-4cd5-b5ed-190987539508 2024-09-22T00:52:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:11 2024-09-22T00:52:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:52:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10' 2024-09-22T00:52:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:52:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:52:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 747e5399-1569-4c2b-9ca7-3e020ff9d82c 2024-09-22T00:52:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:11 2024-09-22T00:52:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:52:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10' 2024-09-22T00:52:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:13.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:52:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:11' 2024-09-22T00:52:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-22 00:52:11' 2024-09-22T00:52:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:52:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:11' 2024-09-22T00:52:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:13.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:52:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e994f3be-a939-4371-b3d8-651240a01b3d 2024-09-22T00:52:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:52:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:11 2024-09-22T00:52:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:52:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:52:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:52:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:10' 2024-09-22T00:52:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:52:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:52:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:52:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:11' 2024-09-22T00:52:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-22 00:52:11' 2024-09-22T00:52:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:52:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:11' 2024-09-22T00:52:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:11' 2024-09-22T00:52:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-22 00:52:11' 2024-09-22T00:52:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:52:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:52:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:11' 2024-09-22T00:52:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10' 2024-09-22T00:52:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:52:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:11' 2024-09-22T00:52:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:52:13.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:13.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:13.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:13.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-22T00:52:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:52:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:52:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-22T00:52:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:52:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:52:13.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:52:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:52:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:52:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:52:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:52:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:52:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-22T00:52:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:52:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:52:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-22T00:52:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:52:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:52:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-22T00:52:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:52:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:52:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-22T00:52:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:52:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:52:13.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-22T00:52:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:52:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:52:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-22T00:52:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10' 2024-09-22T00:52:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:52:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10' 2024-09-22T00:52:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:52:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:11' 2024-09-22T00:52:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:52:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:11cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:52:10cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:52:10' 2024-09-22T00:52:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:52:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:11' 2024-09-22T00:52:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:11' 2024-09-22T00:52:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:52:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:52:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-22T00:52:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:52:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:52:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-22T00:52:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-22T00:52:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:52:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:52:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-22T00:52:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:52:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:52:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:52:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:52:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-22T00:52:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:52:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:52:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-22T00:52:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd2 2024-09-22T00:52:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-22T00:52:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:52:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd1 2024-09-22T00:52:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-22T00:52:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd0 2024-09-22T00:52:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:52:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:52:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-22T00:52:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd4 2024-09-22T00:52:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-22T00:52:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:52:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd3 2024-09-22T00:52:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-22T00:52:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:52:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd5 2024-09-22T00:52:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-22T00:52:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:52:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:52:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:52:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:52:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:52:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:52:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:52:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-22T00:52:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:52:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:52:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:52:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:52:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-22T00:52:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-22T00:52:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:52:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:52:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-22T00:52:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd6 2024-09-22T00:52:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:52:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-22T00:52:15.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd7 2024-09-22T00:52:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd8 2024-09-22T00:52:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:52:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-22T00:52:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:52:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-22T00:52:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd9 2024-09-22T00:52:15.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-22T00:52:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:52:18.194 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:52:18.194+0000 7f54f2d95640 -1 reset not still connected to 0x5617f324f450 2024-09-22T00:52:18.194 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:52:18.194+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281450 2024-09-22T00:52:18.194 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:52:18.194+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281520 2024-09-22T00:52:18.194 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:52:18.194+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:52:18.195 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:52:18.194+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T00:52:18.195 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:52:18.194+0000 7f54f2d95640 -1 reset not still connected to 0x5617f6fb81a0 2024-09-22T00:52:18.195 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:52:18.194+0000 7f54f2d95640 -1 reset not still connected to 0x5617f6fb8b60 2024-09-22T00:52:41.148 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:52:41.144+0000 7fab1f496640 -1 reset not still connected to 0x55fa8207f1e0 2024-09-22T00:52:41.148 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:52:41.144+0000 7fab1f496640 -1 reset not still connected to 0x55fa8207f2b0 2024-09-22T00:52:41.148 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:52:41.144+0000 7fab1f496640 -1 reset not still connected to 0x55fa94c17380 2024-09-22T00:52:41.151 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:52:41.148+0000 7fab1f496640 -1 reset not still connected to 0x55fa94c175f0 2024-09-22T00:52:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=850a824ddf0dcd95a87afb327e4cb8b5 2024-09-22T00:52:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-22T00:52:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=e73fdbd3ca8fdcb6ffae955837fb3b32 2024-09-22T00:52:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-22T00:52:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 850a824ddf0dcd95a87afb327e4cb8b5 != \8\5\0\a\8\2\4\d\d\f\0\d\c\d\9\5\a\8\7\a\f\b\3\2\7\e\4\c\b\8\b\5 ]] 2024-09-22T00:52:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=8a5e4feb46e0e33ce41483fc605ed340 2024-09-22T00:52:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-22T00:52:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ e73fdbd3ca8fdcb6ffae955837fb3b32 != \e\7\3\f\d\b\d\3\c\a\8\f\d\c\b\6\f\f\a\e\9\5\5\8\3\7\f\b\3\b\3\2 ]] 2024-09-22T00:52:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 8a5e4feb46e0e33ce41483fc605ed340 != \8\a\5\e\4\f\e\b\4\6\e\0\e\3\3\c\e\4\1\4\8\3\f\c\6\0\5\e\d\3\4\0 ]] 2024-09-22T00:52:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=1c1ca4960d284b62d13ec703352027d8 2024-09-22T00:52:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-22T00:52:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 1c1ca4960d284b62d13ec703352027d8 != \1\c\1\c\a\4\9\6\0\d\2\8\4\b\6\2\d\1\3\e\c\7\0\3\3\5\2\0\2\7\d\8 ]] 2024-09-22T00:52:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=c6a623473fd18db36a355a5730937b9b 2024-09-22T00:52:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-22T00:52:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ c6a623473fd18db36a355a5730937b9b != \c\6\a\6\2\3\4\7\3\f\d\1\8\d\b\3\6\a\3\5\5\a\5\7\3\0\9\3\7\b\9\b ]] 2024-09-22T00:52:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=0e31c7ceb06486d614f96c9eaf6bfbcc 2024-09-22T00:52:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-22T00:52:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 0e31c7ceb06486d614f96c9eaf6bfbcc != \0\e\3\1\c\7\c\e\b\0\6\4\8\6\d\6\1\4\f\9\6\c\9\e\a\f\6\b\f\b\c\c ]] 2024-09-22T00:52:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=64a3bde46bdc6f043f62b7742096054d 2024-09-22T00:52:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-22T00:52:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 64a3bde46bdc6f043f62b7742096054d != \6\4\a\3\b\d\e\4\6\b\d\c\6\f\0\4\3\f\6\2\b\7\7\4\2\0\9\6\0\5\4\d ]] 2024-09-22T00:52:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:52:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 127925 2024-09-22T00:52:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:52:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 127927 2024-09-22T00:52:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:52:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 127929 2024-09-22T00:52:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=413e91c4573c1a50cb58099ed0d715d2 2024-09-22T00:52:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-22T00:52:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=6b2da8a3960069b45c3ae05ae74347ed 2024-09-22T00:52:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-22T00:52:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 413e91c4573c1a50cb58099ed0d715d2 != \4\1\3\e\9\1\c\4\5\7\3\c\1\a\5\0\c\b\5\8\0\9\9\e\d\0\d\7\1\5\d\2 ]] 2024-09-22T00:52:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 6b2da8a3960069b45c3ae05ae74347ed != \6\b\2\d\a\8\a\3\9\6\0\0\6\9\b\4\5\c\3\a\e\0\5\a\e\7\4\3\4\7\e\d ]] 2024-09-22T00:52:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=d66ac50f2e527babdf05301cb551916e 2024-09-22T00:52:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-22T00:52:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ d66ac50f2e527babdf05301cb551916e != \d\6\6\a\c\5\0\f\2\e\5\2\7\b\a\b\d\f\0\5\3\0\1\c\b\5\5\1\9\1\6\e ]] 2024-09-22T00:52:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:52:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 127931 2024-09-22T00:52:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:52:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 127933 2024-09-22T00:52:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:52:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 127935 2024-09-22T00:52:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:52:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 127937 2024-09-22T00:52:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:52:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 127938 2024-09-22T00:52:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:52:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 127941 2024-09-22T00:52:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:52:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:52:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-22T00:52:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-22T00:52:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:52:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:52:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-22T00:52:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:52:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:52:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-22T00:52:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:52:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:52:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:52:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:52:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 747e5399-1569-4c2b-9ca7-3e020ff9d82c 2024-09-22T00:52:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:52:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":6366822.4,"bytes_per_snapshot":63668224.0,"last_snapshot_bytes":63668224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"} 2024-09-22T00:52:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:35 2024-09-22T00:52:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:52:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:52:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:52:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:45' 2024-09-22T00:52:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:52:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:52:50.421 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":6366822.4,"bytes_per_snapshot":63668224.0,"last_snapshot_bytes":63668224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"} 2024-09-22T00:52:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:52:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:52:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:35' 2024-09-22T00:52:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-22 00:52:35' 2024-09-22T00:52:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:52:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:52:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:35' 2024-09-22T00:52:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1747e5399-1569-4c2b-9ca7-3e020ff9d82cup+replayingreplaying, {"bytes_per_second":6366822.4,"bytes_per_snapshot":63668224.0,"last_snapshot_bytes":63668224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50image-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+replayingreplaying, {"bytes_per_second":6834176.0,"bytes_per_snapshot":68341760.0,"last_snapshot_bytes":68341760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+replayingreplaying, {"bytes_per_second":6659276.8,"bytes_per_snapshot":66592768.0,"last_snapshot_bytes":66592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+replayingreplaying, {"bytes_per_second":6518784.0,"bytes_per_snapshot":65187840.0,"last_snapshot_bytes":65187840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+replayingreplaying, {"bytes_per_second":6484787.2,"bytes_per_snapshot":64847872.0,"last_snapshot_bytes":64847872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+replayingreplaying, {"bytes_per_second":7215513.6,"bytes_per_snapshot":72155136.0,"last_snapshot_bytes":72155136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+replayingreplaying, {"bytes_per_second":6626508.8,"bytes_per_snapshot":66265088.0,"last_snapshot_bytes":66265088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+replayingreplaying, {"bytes_per_second":6157516.8,"bytes_per_snapshot":61575168.0,"last_snapshot_bytes":61575168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+replayingreplaying, {"bytes_per_second":6344294.4,"bytes_per_snapshot":63442944.0,"last_snapshot_bytes":63442944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50image-primary97485789d-5063-4337-b05d-6cd26232c653up+replayingreplaying, {"bytes_per_second":6501171.2,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50' 2024-09-22T00:52:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:52:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:35' 2024-09-22T00:52:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:52:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:52:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:52:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:52:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:52:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-22T00:52:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:52:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-22T00:52:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-22T00:52:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:52:52.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-22T00:52:52.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:52.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:52.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:52:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:52:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary1 2024-09-22T00:52:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary1 2024-09-22T00:52:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:52:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-22T00:52:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-22T00:52:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:52:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:52:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-22T00:52:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:52:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:52:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:53.216 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:52:53.213+0000 7fab5fe19640 -1 reset not still connected to 0x56242145cdd0 2024-09-22T00:52:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-22T00:52:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:52:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:52:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:52:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:52:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8685aec3-047e-4aad-924b-f779ecacae9d 2024-09-22T00:52:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:52:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":6659276.8,"bytes_per_snapshot":66592768.0,"last_snapshot_bytes":66592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"} 2024-09-22T00:52:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:35 2024-09-22T00:52:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:52:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:52:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:52:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:50' 2024-09-22T00:52:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:52:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:52:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":6659276.8,"bytes_per_snapshot":66592768.0,"last_snapshot_bytes":66592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"} 2024-09-22T00:52:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:52:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:52:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:35' 2024-09-22T00:52:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-22 00:52:35' 2024-09-22T00:52:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:52:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:52:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:35' 2024-09-22T00:52:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK942614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+replayingreplaying, {"bytes_per_second":6834176.0,"bytes_per_snapshot":68341760.0,"last_snapshot_bytes":68341760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50image-primary28685aec3-047e-4aad-924b-f779ecacae9dup+replayingreplaying, {"bytes_per_second":6659276.8,"bytes_per_snapshot":66592768.0,"last_snapshot_bytes":66592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+replayingreplaying, {"bytes_per_second":6518784.0,"bytes_per_snapshot":65187840.0,"last_snapshot_bytes":65187840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+replayingreplaying, {"bytes_per_second":6484787.2,"bytes_per_snapshot":64847872.0,"last_snapshot_bytes":64847872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+replayingreplaying, {"bytes_per_second":7215513.6,"bytes_per_snapshot":72155136.0,"last_snapshot_bytes":72155136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+replayingreplaying, {"bytes_per_second":6626508.8,"bytes_per_snapshot":66265088.0,"last_snapshot_bytes":66265088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+replayingreplaying, {"bytes_per_second":6157516.8,"bytes_per_snapshot":61575168.0,"last_snapshot_bytes":61575168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+replayingreplaying, {"bytes_per_second":6344294.4,"bytes_per_snapshot":63442944.0,"last_snapshot_bytes":63442944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50image-primary97485789d-5063-4337-b05d-6cd26232c653up+replayingreplaying, {"bytes_per_second":6501171.2,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:50' 2024-09-22T00:52:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:52:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:35' 2024-09-22T00:52:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:52:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:52:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:52:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:52:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:52:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-22T00:52:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:52:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-22T00:52:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-22T00:52:54.261 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:52:54.257+0000 7f54f2d95640 -1 reset not still connected to 0x5617f2467380 2024-09-22T00:52:54.261 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:52:54.257+0000 7f54f2d95640 -1 reset not still connected to 0x5617f324f380 2024-09-22T00:52:54.262 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:52:54.257+0000 7f54f2d95640 -1 reset not still connected to 0x5617f324f450 2024-09-22T00:52:54.262 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:52:54.257+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281450 2024-09-22T00:52:54.262 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:52:54.257+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281520 2024-09-22T00:52:54.262 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:52:54.257+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:52:54.262 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:52:54.257+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T00:52:54.262 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:52:54.257+0000 7f54f2d95640 -1 reset not still connected to 0x5617f6fb81a0 2024-09-22T00:52:54.262 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:52:54.257+0000 7f54f2d95640 -1 reset not still connected to 0x5617f6fb8b60 2024-09-22T00:52:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:52:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-22T00:52:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:52:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:52:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary2 2024-09-22T00:52:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary2 2024-09-22T00:52:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:52:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-22T00:52:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-22T00:52:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:52:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:52:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-22T00:52:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:52:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:52:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-22T00:52:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:52:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:52:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:52:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:52:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 951a197c-f95a-4cd5-b5ed-190987539508 2024-09-22T00:52:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:52:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":6518784.0,"bytes_per_snapshot":65187840.0,"last_snapshot_bytes":65187840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"} 2024-09-22T00:52:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:52:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:35 2024-09-22T00:52:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:52:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:52:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:52:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:52:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:55' 2024-09-22T00:52:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:52:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:52:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:52:57.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":6518784.0,"bytes_per_snapshot":65187840.0,"last_snapshot_bytes":65187840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"} 2024-09-22T00:52:57.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:52:57.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:52:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:52:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:52:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:52:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:52:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:52:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:52:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:35' 2024-09-22T00:52:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-22 00:52:35' 2024-09-22T00:52:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:52:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:52:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:35' 2024-09-22T00:52:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:52:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK842614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+replayingreplaying, {"bytes_per_second":6834176.0,"bytes_per_snapshot":68341760.0,"last_snapshot_bytes":68341760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:55image-primary3951a197c-f95a-4cd5-b5ed-190987539508up+replayingreplaying, {"bytes_per_second":6518784.0,"bytes_per_snapshot":65187840.0,"last_snapshot_bytes":65187840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:55image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+replayingreplaying, {"bytes_per_second":6484787.2,"bytes_per_snapshot":64847872.0,"last_snapshot_bytes":64847872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:55image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+replayingreplaying, {"bytes_per_second":7215513.6,"bytes_per_snapshot":72155136.0,"last_snapshot_bytes":72155136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:55image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+replayingreplaying, {"bytes_per_second":6626508.8,"bytes_per_snapshot":66265088.0,"last_snapshot_bytes":66265088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:55image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+replayingreplaying, {"bytes_per_second":6157516.8,"bytes_per_snapshot":61575168.0,"last_snapshot_bytes":61575168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:55image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+replayingreplaying, {"bytes_per_second":6344294.4,"bytes_per_snapshot":63442944.0,"last_snapshot_bytes":63442944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:55image-primary97485789d-5063-4337-b05d-6cd26232c653up+replayingreplaying, {"bytes_per_second":6501171.2,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:55' 2024-09-22T00:52:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:52:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:52:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:35' 2024-09-22T00:52:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:52:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:52:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:52:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:52:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:52:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-22T00:52:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:52:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:52:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-22T00:52:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-22T00:52:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:52:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-22T00:52:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:52:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary3 2024-09-22T00:52:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:52:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:52:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary3 2024-09-22T00:52:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:52:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-22T00:52:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-22T00:52:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:52:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:52:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-22T00:52:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:52:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:52:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:52:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:52:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-22T00:52:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:52:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:52:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:52:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:52:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:52:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:52:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:52:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:52:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:53:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:53:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6a888d77-180b-47c1-9533-64e2804f0e56 2024-09-22T00:53:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:53:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":6484787.2,"bytes_per_snapshot":64847872.0,"last_snapshot_bytes":64847872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"} 2024-09-22T00:53:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:53:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:35 2024-09-22T00:53:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:53:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:53:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:53:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:53:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:55' 2024-09-22T00:53:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:53:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:53:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:53:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":6484787.2,"bytes_per_snapshot":64847872.0,"last_snapshot_bytes":64847872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"} 2024-09-22T00:53:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:53:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:53:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:53:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:53:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:53:00.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:53:00.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:53:00.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:53:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:35' 2024-09-22T00:53:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-22 00:52:35' 2024-09-22T00:53:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:53:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:35' 2024-09-22T00:53:00.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:53:00.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:53:00.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:53:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK742614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+replayingreplaying, {"bytes_per_second":6834176.0,"bytes_per_snapshot":68341760.0,"last_snapshot_bytes":68341760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:55image-primary46a888d77-180b-47c1-9533-64e2804f0e56up+replayingreplaying, {"bytes_per_second":6484787.2,"bytes_per_snapshot":64847872.0,"last_snapshot_bytes":64847872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:55image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+replayingreplaying, {"bytes_per_second":7215513.6,"bytes_per_snapshot":72155136.0,"last_snapshot_bytes":72155136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:55image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+replayingreplaying, {"bytes_per_second":6626508.8,"bytes_per_snapshot":66265088.0,"last_snapshot_bytes":66265088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:55image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+replayingreplaying, {"bytes_per_second":6157516.8,"bytes_per_snapshot":61575168.0,"last_snapshot_bytes":61575168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:55image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+replayingreplaying, {"bytes_per_second":6344294.4,"bytes_per_snapshot":63442944.0,"last_snapshot_bytes":63442944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:55image-primary97485789d-5063-4337-b05d-6cd26232c653up+replayingreplaying, {"bytes_per_second":6501171.2,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:52:55' 2024-09-22T00:53:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:53:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:53:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:35' 2024-09-22T00:53:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:53:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:53:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:53:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:53:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:53:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-22T00:53:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:53:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:53:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-22T00:53:00.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-22T00:53:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:53:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-22T00:53:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:53:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:53:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:53:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary4 2024-09-22T00:53:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary4 2024-09-22T00:53:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:53:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-22T00:53:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-22T00:53:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:53:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:53:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-22T00:53:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:53:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:53:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:53:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:53:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:53:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-22T00:53:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:53:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:53:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:53:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:53:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:53:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:53:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:53:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: e994f3be-a939-4371-b3d8-651240a01b3d 2024-09-22T00:53:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:53:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":7215513.6,"bytes_per_snapshot":72155136.0,"last_snapshot_bytes":72155136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"} 2024-09-22T00:53:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:53:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:52:35 2024-09-22T00:53:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:53:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:53:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:53:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:53:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:53:00' 2024-09-22T00:53:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:53:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:53:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:53:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":7215513.6,"bytes_per_snapshot":72155136.0,"last_snapshot_bytes":72155136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"} 2024-09-22T00:53:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:53:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:53:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:53:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:53:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:53:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:53:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:53:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:53:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:35' 2024-09-22T00:53:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-22 00:52:35' 2024-09-22T00:53:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:53:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:52:35' 2024-09-22T00:53:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:53:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:53:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:53:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK642614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+replayingreplaying, {"bytes_per_second":6834176.0,"bytes_per_snapshot":68341760.0,"last_snapshot_bytes":68341760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:00image-primary5e994f3be-a939-4371-b3d8-651240a01b3dup+replayingreplaying, {"bytes_per_second":7215513.6,"bytes_per_snapshot":72155136.0,"last_snapshot_bytes":72155136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966334,"remote_snapshot_timestamp":1726966334,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:00image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+replayingreplaying, {"bytes_per_second":6626508.8,"bytes_per_snapshot":66265088.0,"last_snapshot_bytes":66265088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:00image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+replayingreplaying, {"bytes_per_second":6157516.8,"bytes_per_snapshot":61575168.0,"last_snapshot_bytes":61575168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:00image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+replayingreplaying, {"bytes_per_second":6344294.4,"bytes_per_snapshot":63442944.0,"last_snapshot_bytes":63442944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:00image-primary97485789d-5063-4337-b05d-6cd26232c653up+replayingreplaying, {"bytes_per_second":6501171.2,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:52:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:00' 2024-09-22T00:53:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:53:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:53:03.259 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:52:35' 2024-09-22T00:53:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:53:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:53:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:53:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:53:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:53:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-22T00:53:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:53:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:53:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-22T00:53:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-22T00:53:04.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:53:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-22T00:53:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:53:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:53:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:53:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary5 2024-09-22T00:53:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary5 2024-09-22T00:53:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:53:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-22T00:53:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-22T00:53:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:53:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:53:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-22T00:53:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:53:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:53:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:53:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:53:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:53:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-22T00:53:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:53:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:53:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:53:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:53:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:53:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:53:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:53:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6334f7b1-bf34-4d11-af93-7a63cb657782 2024-09-22T00:53:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:53:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66265088.0,"last_snapshot_bytes":66265088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"} 2024-09-22T00:53:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:53:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:53:05 2024-09-22T00:53:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:53:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:53:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:53:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:53:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:53:05' 2024-09-22T00:53:06.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:53:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:53:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:53:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66265088.0,"last_snapshot_bytes":66265088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"} 2024-09-22T00:53:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:53:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:53:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:53:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:53:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:53:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:53:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:53:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:53:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:53:05' 2024-09-22T00:53:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-22 00:53:05' 2024-09-22T00:53:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:53:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:53:05' 2024-09-22T00:53:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:53:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:53:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:53:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK542614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":68341760.0,"last_snapshot_bytes":68341760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:53:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:05image-primary66334f7b1-bf34-4d11-af93-7a63cb657782up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66265088.0,"last_snapshot_bytes":66265088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:53:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:05image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61575168.0,"last_snapshot_bytes":61575168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:53:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:05image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63442944.0,"last_snapshot_bytes":63442944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:53:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:05image-primary97485789d-5063-4337-b05d-6cd26232c653up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:53:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:05' 2024-09-22T00:53:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:53:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:53:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:53:05' 2024-09-22T00:53:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:53:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:53:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:53:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:53:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:53:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-22T00:53:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:53:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:53:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-22T00:53:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-22T00:53:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:53:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-22T00:53:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:53:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:53:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:53:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary6 2024-09-22T00:53:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary6 2024-09-22T00:53:08.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:53:08.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-22T00:53:08.003 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-22T00:53:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:53:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:53:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-22T00:53:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:53:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:53:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:53:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:53:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:53:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-22T00:53:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:53:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:53:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:53:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:53:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:53:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:53:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:53:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 236d5724-41f0-4963-b51e-ce5af53eca01 2024-09-22T00:53:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:53:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61575168.0,"last_snapshot_bytes":61575168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"} 2024-09-22T00:53:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:53:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:53:05 2024-09-22T00:53:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:53:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:53:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:53:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:53:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:53:05' 2024-09-22T00:53:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:53:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:53:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:53:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61575168.0,"last_snapshot_bytes":61575168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"} 2024-09-22T00:53:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:53:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:53:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:53:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:53:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:53:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:53:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:53:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:53:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:53:05' 2024-09-22T00:53:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-22 00:53:05' 2024-09-22T00:53:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:53:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:53:05' 2024-09-22T00:53:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:53:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:53:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:53:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK442614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":68341760.0,"last_snapshot_bytes":68341760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:53:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:05image-primary7236d5724-41f0-4963-b51e-ce5af53eca01up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61575168.0,"last_snapshot_bytes":61575168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:53:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:05image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63442944.0,"last_snapshot_bytes":63442944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:53:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:05image-primary97485789d-5063-4337-b05d-6cd26232c653up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:53:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:05' 2024-09-22T00:53:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:53:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:53:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:53:05' 2024-09-22T00:53:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:53:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:53:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:53:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:53:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:53:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-22T00:53:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:53:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:53:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-22T00:53:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-22T00:53:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:53:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-22T00:53:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:53:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:53:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:53:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary7 2024-09-22T00:53:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary7 2024-09-22T00:53:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:53:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-22T00:53:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-22T00:53:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:53:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:53:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-22T00:53:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:53:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:53:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:53:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:53:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:53:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-22T00:53:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:53:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:53:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:53:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:53:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:53:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:53:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:53:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 054e263e-3a54-42f0-bae1-57f67ead0eff 2024-09-22T00:53:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:53:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63442944.0,"last_snapshot_bytes":63442944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"} 2024-09-22T00:53:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:53:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:53:05 2024-09-22T00:53:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:53:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:53:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:53:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:53:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:53:10' 2024-09-22T00:53:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:53:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:53:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:53:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63442944.0,"last_snapshot_bytes":63442944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"} 2024-09-22T00:53:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:53:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:53:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:53:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:53:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:53:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:53:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:53:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:53:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:53:05' 2024-09-22T00:53:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-22 00:53:05' 2024-09-22T00:53:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:53:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:53:05' 2024-09-22T00:53:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:53:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:53:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:53:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":68341760.0,"last_snapshot_bytes":68341760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:53:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:10image-primary8054e263e-3a54-42f0-bae1-57f67ead0effup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63442944.0,"last_snapshot_bytes":63442944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:53:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:53:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:10' 2024-09-22T00:53:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:53:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:53:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:53:05' 2024-09-22T00:53:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:53:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:53:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:53:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:53:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:53:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:12.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:12.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-22T00:53:12.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:53:12.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:12.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:53:12.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-22T00:53:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-22T00:53:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:53:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-22T00:53:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:53:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:53:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:53:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary8 2024-09-22T00:53:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary8 2024-09-22T00:53:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:53:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-22T00:53:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-22T00:53:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:53:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:53:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-22T00:53:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:53:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:53:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:53:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:53:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:53:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-22T00:53:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:53:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:53:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:53:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:53:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:53:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:53:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:53:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 7485789d-5063-4337-b05d-6cd26232c653 2024-09-22T00:53:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:53:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"} 2024-09-22T00:53:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:53:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:53:05 2024-09-22T00:53:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:53:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:53:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:53:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:53:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:53:10' 2024-09-22T00:53:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:53:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:53:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:53:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"} 2024-09-22T00:53:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:53:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:53:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:53:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:53:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:53:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:53:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:53:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:53:15.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:53:05' 2024-09-22T00:53:15.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-22 00:53:05' 2024-09-22T00:53:15.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:53:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:53:05' 2024-09-22T00:53:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:53:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:53:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:53:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK242614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":68341760.0,"last_snapshot_bytes":68341760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:53:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:10image-primary97485789d-5063-4337-b05d-6cd26232c653up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:53:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:10' 2024-09-22T00:53:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:53:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:53:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:53:05' 2024-09-22T00:53:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:53:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:53:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:53:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:53:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:53:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-22T00:53:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:53:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:53:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-22T00:53:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-22T00:53:15.538 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:53:15.541+0000 7fab5fe19640 -1 reset not still connected to 0x5623f97bd860 2024-09-22T00:53:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:53:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-22T00:53:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:53:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:53:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:53:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary9 2024-09-22T00:53:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary9 2024-09-22T00:53:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:53:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-22T00:53:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-22T00:53:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:53:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:53:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-22T00:53:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:53:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:53:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:53:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:53:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:53:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-22T00:53:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:53:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:53:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:53:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:53:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:53:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:53:18.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:53:18.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: af2421d2-f6c8-49a0-8e4a-53d7470b473c 2024-09-22T00:53:18.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:53:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":68341760.0,"last_snapshot_bytes":68341760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"} 2024-09-22T00:53:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:53:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:53:05 2024-09-22T00:53:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:53:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:53:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:53:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:53:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:53:15' 2024-09-22T00:53:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:53:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:53:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:53:18.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":68341760.0,"last_snapshot_bytes":68341760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"} 2024-09-22T00:53:18.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:53:18.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:53:18.062 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:53:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:53:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:53:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:53:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:53:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:53:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:53:05' 2024-09-22T00:53:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-22 00:53:05' 2024-09-22T00:53:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:53:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:53:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:53:05' 2024-09-22T00:53:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:53:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:53:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:53:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK142614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10af2421d2-f6c8-49a0-8e4a-53d7470b473cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":68341760.0,"last_snapshot_bytes":68341760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966333,"remote_snapshot_timestamp":1726966333,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:53:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:53:15' 2024-09-22T00:53:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:53:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:53:18.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:53:05' 2024-09-22T00:53:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:53:18.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:53:18.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:53:18.181 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:53:18.182 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:53:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-22T00:53:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:53:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:53:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-22T00:53:18.242 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-22T00:53:19.287 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:53:19.285+0000 7fab5fe19640 -1 reset not still connected to 0x5623f97bd860 2024-09-22T00:53:19.287 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:53:19.285+0000 7fab5fe19640 -1 reset not still connected to 0x5623f97bdba0 2024-09-22T00:53:19.287 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:53:19.285+0000 7fab5fe19640 -1 reset not still connected to 0x562426c3a750 2024-09-22T00:53:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:53:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-22T00:53:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:53:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:53:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:53:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary10 2024-09-22T00:53:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary10 2024-09-22T00:53:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..10} 2024-09-22T00:53:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS=() 2024-09-22T00:53:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS=() 2024-09-22T00:53:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS=() 2024-09-22T00:53:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS=() 2024-09-22T00:53:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ RET=0 2024-09-22T00:53:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:53:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-22T00:53:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary1 2024-09-22T00:53:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-22T00:53:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:53:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:53:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-22T00:53:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:53:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:53:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:53:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:53:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-22T00:53:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:53:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:53:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-22T00:53:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:53:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-22T00:53:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:53:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-22T00:53:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary1': 2024-09-22T00:53:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:53:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:53:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:53:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 38c31ee8dd1a 2024-09-22T00:53:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.38c31ee8dd1a 2024-09-22T00:53:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:53:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:53:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:53:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:53:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:53:20 2024 2024-09-22T00:53:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:53:20 2024 2024-09-22T00:53:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:53:20 2024 2024-09-22T00:53:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:53:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:53:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 0f6d12b8-9468-4767-b8f3-f62a5392a241 2024-09-22T00:53:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:53:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:53:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:53:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-22T00:53:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd0 2024-09-22T00:53:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:53:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-22T00:53:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:53:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:53:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:53:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 8d21d28b-d71f-424d-9317-0b03973f238e 2024-09-22T00:53:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:53:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:53:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:53:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:53:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:53:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:53:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary1 2024-09-22T00:53:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-22T00:53:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:53:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary1 2024-09-22T00:53:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:53:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-22T00:53:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:53:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:53:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:53:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-22T00:53:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary2 2024-09-22T00:53:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:53:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-22T00:53:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:53:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:53:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:53:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:53:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-22T00:53:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:53:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:53:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary1 2024-09-22T00:53:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:53:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:53:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-22T00:53:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary1 2024-09-22T00:53:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-22T00:53:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:53:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:53:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-22T00:53:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:53:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-22T00:53:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2996 2024-09-22T00:53:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:53:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-22T00:53:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary2': 2024-09-22T00:53:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:53:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:53:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:53:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 38d24fe42059 2024-09-22T00:53:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.38d24fe42059 2024-09-22T00:53:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:53:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:53:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:53:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:53:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:53:21 2024 2024-09-22T00:53:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:53:21 2024 2024-09-22T00:53:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:53:21 2024 2024-09-22T00:53:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:53:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:53:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: c7a13db2-2e62-40a8-86e3-e19844d44822 2024-09-22T00:53:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:53:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:53:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:53:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-22T00:53:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd1 2024-09-22T00:53:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:53:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-22T00:53:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:53:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:53:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:53:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 653f328a-79ba-4491-9743-cba8bdc304d1 2024-09-22T00:53:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:53:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:53:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:53:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:53:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:53:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:53:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary2 2024-09-22T00:53:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-22T00:53:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:53:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary2 2024-09-22T00:53:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:53:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-22T00:53:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:53:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:53:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:53:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary2 2024-09-22T00:53:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-22T00:53:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary3 2024-09-22T00:53:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:53:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-22T00:53:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:53:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:53:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary2 2024-09-22T00:53:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-22T00:53:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:53:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:53:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-22T00:53:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:53:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:53:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:53:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:53:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-22T00:53:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:53:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:53:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-22T00:53:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:53:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-22T00:53:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:53:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-22T00:53:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:53:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-22T00:53:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:53:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 2998 2024-09-22T00:53:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-22T00:53:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary3': 2024-09-22T00:53:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:53:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:53:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:53:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 38e04ad21654 2024-09-22T00:53:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.38e04ad21654 2024-09-22T00:53:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:53:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:53:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:53:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:53:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:53:22 2024 2024-09-22T00:53:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:53:22 2024 2024-09-22T00:53:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:53:22 2024 2024-09-22T00:53:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:53:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:53:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 6175a8d9-0e7f-4d09-9b55-cc3a19b9272f 2024-09-22T00:53:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:53:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:53:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:53:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-22T00:53:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd2 2024-09-22T00:53:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:53:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-22T00:53:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:53:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:53:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:53:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: f41296dc-b895-48c6-a8e8-33a27fbd0a82 2024-09-22T00:53:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:53:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:53:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:53:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:53:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:53:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:53:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary3 2024-09-22T00:53:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-22T00:53:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:53:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary3 2024-09-22T00:53:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:53:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-22T00:53:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:53:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:53:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:53:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:53:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:53:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:53:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary3 2024-09-22T00:53:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-22T00:53:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary4 2024-09-22T00:53:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-22T00:53:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary3 2024-09-22T00:53:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-22T00:53:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:53:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:53:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-22T00:53:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:53:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:53:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:53:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:53:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-22T00:53:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:53:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:53:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-22T00:53:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:53:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-22T00:53:23.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:53:23.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-22T00:53:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:53:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3000 2024-09-22T00:53:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-22T00:53:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:24.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary4': 2024-09-22T00:53:24.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:53:24.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:53:24.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:53:24.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 38eea1df31a1 2024-09-22T00:53:24.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.38eea1df31a1 2024-09-22T00:53:24.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:53:24.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:53:24.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:53:24.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:53:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:53:23 2024 2024-09-22T00:53:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:53:23 2024 2024-09-22T00:53:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:53:23 2024 2024-09-22T00:53:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:53:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:53:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: d8084146-8956-435b-a81f-43c80a014180 2024-09-22T00:53:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:53:24.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:53:24.008 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:53:24.009 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-22T00:53:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd3 2024-09-22T00:53:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:53:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-22T00:53:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:53:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:53:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:53:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 74745564-4b63-4ce1-852a-ac554af3543f 2024-09-22T00:53:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:53:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:53:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:53:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:53:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:53:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:53:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary4 2024-09-22T00:53:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-22T00:53:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:53:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary4 2024-09-22T00:53:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:53:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-22T00:53:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:53:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:53:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:53:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:53:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:53:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:53:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-22T00:53:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary5 2024-09-22T00:53:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-22T00:53:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary4 2024-09-22T00:53:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:53:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:53:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-22T00:53:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:53:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:53:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:53:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:53:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary4 2024-09-22T00:53:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-22T00:53:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-22T00:53:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:53:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:53:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-22T00:53:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:53:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-22T00:53:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:53:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-22T00:53:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:53:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-22T00:53:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3002 2024-09-22T00:53:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary5': 2024-09-22T00:53:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:53:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:53:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:53:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 38fcf4fab5d4 2024-09-22T00:53:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.38fcf4fab5d4 2024-09-22T00:53:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:53:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:53:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:53:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:53:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:53:24 2024 2024-09-22T00:53:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:53:24 2024 2024-09-22T00:53:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:53:24 2024 2024-09-22T00:53:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:53:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:53:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 199f939a-c91f-4933-85bd-fed08290fd6e 2024-09-22T00:53:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:53:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:53:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:53:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-22T00:53:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:53:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:53:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:53:24.990 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:24.989+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281520 2024-09-22T00:53:24.990 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:24.989+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:53:24.996 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:24.997+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T00:53:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd4 2024-09-22T00:53:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:53:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-22T00:53:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:53:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:53:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:53:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 6ce0ebce-5ff9-4938-81bf-17b411581487 2024-09-22T00:53:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:53:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:53:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:53:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:53:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:53:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:53:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary5 2024-09-22T00:53:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-22T00:53:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:53:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary5 2024-09-22T00:53:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:53:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-22T00:53:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:53:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:53:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary5 2024-09-22T00:53:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:53:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:53:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-22T00:53:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:53:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary6 2024-09-22T00:53:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:53:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-22T00:53:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary5 2024-09-22T00:53:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-22T00:53:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:53:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:53:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-22T00:53:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:53:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:53:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:53:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:53:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-22T00:53:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:53:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:53:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-22T00:53:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:53:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-22T00:53:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:53:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-22T00:53:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:53:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3005 2024-09-22T00:53:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:53:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:53:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:53:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-22T00:53:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:25.985 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:53:25.985+0000 7fab5fe19640 -1 reset not still connected to 0x5623f97bd860 2024-09-22T00:53:25.985 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:53:25.985+0000 7fab5fe19640 -1 reset not still connected to 0x562401756820 2024-09-22T00:53:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary6': 2024-09-22T00:53:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:53:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:53:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:53:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 390dee4aff60 2024-09-22T00:53:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.390dee4aff60 2024-09-22T00:53:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:53:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:53:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:53:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:53:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:53:25 2024 2024-09-22T00:53:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:53:25 2024 2024-09-22T00:53:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:53:25 2024 2024-09-22T00:53:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:53:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:53:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 4de603dc-70a4-4d8e-ac7a-d2f59bde04b9 2024-09-22T00:53:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:53:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:53:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:53:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-22T00:53:26.170 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:26.169+0000 7f54f2d95640 -1 reset not still connected to 0x5617f324f450 2024-09-22T00:53:26.171 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:26.169+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281520 2024-09-22T00:53:26.171 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:26.169+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:53:26.171 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:26.173+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T00:53:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd5 2024-09-22T00:53:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:53:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-22T00:53:26.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:53:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:53:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:53:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 68d161a6-a80c-444c-b5b7-95f31c845976 2024-09-22T00:53:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:53:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:53:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:53:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:53:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:53:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:53:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:26.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary6 2024-09-22T00:53:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-22T00:53:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:53:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary6 2024-09-22T00:53:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:53:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:53:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:53:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-22T00:53:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-22T00:53:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary7 2024-09-22T00:53:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-22T00:53:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary6 2024-09-22T00:53:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:53:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:53:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-22T00:53:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:53:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:53:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:53:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:53:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary6 2024-09-22T00:53:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:53:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:53:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-22T00:53:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-22T00:53:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:53:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:53:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:53:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:53:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-22T00:53:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:26.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:26.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:53:26.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:26.479 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-22T00:53:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:53:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-22T00:53:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3004 2024-09-22T00:53:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:53:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3008 2024-09-22T00:53:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-22T00:53:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:53:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:53:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:53:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary7': 2024-09-22T00:53:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:53:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:53:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:53:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 391d62d24a3c 2024-09-22T00:53:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.391d62d24a3c 2024-09-22T00:53:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:53:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:53:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:53:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:53:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:53:26 2024 2024-09-22T00:53:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:53:26 2024 2024-09-22T00:53:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:53:26 2024 2024-09-22T00:53:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:53:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:53:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: ec869a35-b7df-4a94-a692-4cf4ac04e207 2024-09-22T00:53:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:53:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:53:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:53:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-22T00:53:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd6 2024-09-22T00:53:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:53:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-22T00:53:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:53:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:53:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:53:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 999ba498-08fc-4636-9d2b-b9d01c7ca3df 2024-09-22T00:53:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:53:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:53:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:53:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:53:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:53:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:53:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary7 2024-09-22T00:53:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-22T00:53:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:53:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary7 2024-09-22T00:53:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:53:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-22T00:53:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:53:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary7 2024-09-22T00:53:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:53:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-22T00:53:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary8 2024-09-22T00:53:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-22T00:53:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:53:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary7 2024-09-22T00:53:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-22T00:53:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:53:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:53:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:53:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:53:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-22T00:53:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:53:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:53:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:53:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:53:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:53:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-22T00:53:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:53:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:53:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-22T00:53:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:53:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-22T00:53:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:53:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-22T00:53:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:53:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:53:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:53:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3007 2024-09-22T00:53:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:53:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3011 2024-09-22T00:53:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-22T00:53:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary8': 2024-09-22T00:53:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:53:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:53:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:53:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 392ebca8c8ad 2024-09-22T00:53:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.392ebca8c8ad 2024-09-22T00:53:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:53:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:53:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:53:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:53:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:53:27 2024 2024-09-22T00:53:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:53:27 2024 2024-09-22T00:53:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:53:27 2024 2024-09-22T00:53:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:53:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:53:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 078a49ba-3e14-42c8-acc1-269d8a4f54d9 2024-09-22T00:53:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:53:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:53:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:53:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-22T00:53:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd7 2024-09-22T00:53:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:53:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-22T00:53:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:53:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:53:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:53:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 1f886b9c-dc9f-499d-9f6b-fee25320d3a8 2024-09-22T00:53:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:53:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:53:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:53:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:53:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:53:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:53:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary8 2024-09-22T00:53:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-22T00:53:28.363 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:53:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary8 2024-09-22T00:53:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:53:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-22T00:53:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:53:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:53:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:53:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:53:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:53:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-22T00:53:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary9 2024-09-22T00:53:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:53:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary8 2024-09-22T00:53:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-22T00:53:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:53:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:53:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-22T00:53:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:53:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary8 2024-09-22T00:53:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:53:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:53:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:53:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-22T00:53:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-22T00:53:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:53:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:53:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-22T00:53:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:53:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-22T00:53:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:53:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-22T00:53:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3010 2024-09-22T00:53:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:53:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:53:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:53:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:53:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3014 2024-09-22T00:53:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-22T00:53:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary9': 2024-09-22T00:53:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:53:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:53:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:53:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 393ec76c44cf 2024-09-22T00:53:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.393ec76c44cf 2024-09-22T00:53:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:53:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:53:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:53:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:53:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:53:28 2024 2024-09-22T00:53:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:53:28 2024 2024-09-22T00:53:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:53:28 2024 2024-09-22T00:53:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:53:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:53:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 8102f56b-719c-4239-9d1a-177deba63ef3 2024-09-22T00:53:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:53:29.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:53:29.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:53:29.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-22T00:53:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd8 2024-09-22T00:53:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:53:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-22T00:53:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:53:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:53:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:53:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: e8157bf1-4340-49f3-a105-dfda9fd1b01d 2024-09-22T00:53:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:53:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:53:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:53:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:53:29.203 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:53:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:53:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:29.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary9 2024-09-22T00:53:29.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-22T00:53:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:53:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary9 2024-09-22T00:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-22T00:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary9 2024-09-22T00:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-22T00:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ MNTPT=test-primary10 2024-09-22T00:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-22T00:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary9 2024-09-22T00:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-22T00:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:53:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n snapshot ']' 2024-09-22T00:53:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-22T00:53:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:53:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local size=128 2024-09-22T00:53:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 10G ']' 2024-09-22T00:53:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ size=10G 2024-09-22T00:53:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ shift 2024-09-22T00:53:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-22T00:53:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ is_pool_mirror_mode_image mirror 2024-09-22T00:53:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' image = image ']' 2024-09-22T00:53:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:53:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-22T00:53:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:53:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mode=snapshot 2024-09-22T00:53:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-22T00:53:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:53:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-22T00:53:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3013 2024-09-22T00:53:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3017 2024-09-22T00:53:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Mirroring enabled 2024-09-22T00:53:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-22T00:53:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:53:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:53:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:53:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:53:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:53:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:53:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:rbd image 'image-primary10': 2024-09-22T00:53:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: size 10 GiB in 2560 objects 2024-09-22T00:53:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: order 22 (4 MiB objects) 2024-09-22T00:53:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: snapshot_count: 1 2024-09-22T00:53:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: id: 39514599b451 2024-09-22T00:53:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: block_name_prefix: rbd_data.39514599b451 2024-09-22T00:53:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: format: 2 2024-09-22T00:53:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-22T00:53:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: op_features: 2024-09-22T00:53:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: flags: 2024-09-22T00:53:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: create_timestamp: Sun Sep 22 00:53:29 2024 2024-09-22T00:53:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: access_timestamp: Sun Sep 22 00:53:29 2024 2024-09-22T00:53:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: modify_timestamp: Sun Sep 22 00:53:29 2024 2024-09-22T00:53:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring state: enabled 2024-09-22T00:53:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring mode: snapshot 2024-09-22T00:53:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring global id: 86d5828f-ec5b-4c64-81bd-fc524d8f94cf 2024-09-22T00:53:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: mirroring primary: true 2024-09-22T00:53:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:53:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:53:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-22T00:53:30.022 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:53:30.020+0000 7fab1f496640 -1 reset not still connected to 0x55fa66b985b0 2024-09-22T00:53:30.022 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:53:30.020+0000 7fab1f496640 -1 reset not still connected to 0x55fa728676c0 2024-09-22T00:53:30.022 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:53:30.020+0000 7fab1f496640 -1 reset not still connected to 0x55fa80ea9040 2024-09-22T00:53:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEV=/dev/rbd9 2024-09-22T00:53:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ DEVS+=($DEV) 2024-09-22T00:53:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-22T00:53:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-22T00:53:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Discarding device blocks: 0/2621440 done 2024-09-22T00:53:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-22T00:53:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Filesystem UUID: 4997df77-78df-42dc-b54d-559eda3025ee 2024-09-22T00:53:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Superblock backups stored on blocks: 2024-09-22T00:53:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-22T00:53:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Allocating group tables: 0/80 done 2024-09-22T00:53:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing inode tables: 0/80 done 2024-09-22T00:53:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Creating journal (16384 blocks): done 2024-09-22T00:53:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-22T00:53:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: 2024-09-22T00:53:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mkdir test-primary10 2024-09-22T00:53:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-22T00:53:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ whoami 2024-09-22T00:53:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo chown ubuntu test-primary10 2024-09-22T00:53:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ SNAP_PIDS+=($!) 2024-09-22T00:53:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-22T00:53:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-22T00:53:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:53:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ slow_untar_workload test-primary10 2024-09-22T00:53:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 131561 2024-09-22T00:53:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mountpt=test-primary10 2024-09-22T00:53:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-22T00:53:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:53:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:53:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:30.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:30.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:53:30.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:53:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local ret=0 2024-09-22T00:53:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-22T00:53:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3016 2024-09-22T00:53:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3021 2024-09-22T00:53:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:53:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:53:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:53:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:53:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3019 2024-09-22T00:53:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3020 2024-09-22T00:53:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:53:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:53:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:53:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:53:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:53:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:53:32.212 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:32.201+0000 7f54f2d95640 -1 reset not still connected to 0x56180602eb60 2024-09-22T00:53:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:53:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:53:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:53:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:53:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:53:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:53:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3022 2024-09-22T00:53:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3023 2024-09-22T00:53:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3024 2024-09-22T00:53:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3025 2024-09-22T00:53:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:53:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:53:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:53:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:53:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:53:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:53:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3026 2024-09-22T00:53:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3027 2024-09-22T00:53:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:53:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:53:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:53:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:53:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:53:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:53:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3029 2024-09-22T00:53:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3028 2024-09-22T00:53:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:53:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:53:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:53:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:53:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:53:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:53:36.009 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:53:36.008+0000 7fab1f496640 -1 reset not still connected to 0x55fa665760d0 2024-09-22T00:53:36.015 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:53:36.008+0000 7fab1f496640 -1 reset not still connected to 0x55fa66576410 2024-09-22T00:53:36.015 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:53:36.008+0000 7fab1f496640 -1 reset not still connected to 0x55fa66b985b0 2024-09-22T00:53:36.015 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:53:36.008+0000 7fab1f496640 -1 reset not still connected to 0x55fa6d9a68f0 2024-09-22T00:53:36.031 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:53:36.024+0000 7fab1f496640 -1 reset not still connected to 0x55fa728676c0 2024-09-22T00:53:36.031 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:53:36.024+0000 7fab1f496640 -1 reset not still connected to 0x55fa80ea9040 2024-09-22T00:53:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3030 2024-09-22T00:53:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3031 2024-09-22T00:53:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:53:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:53:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:53:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:53:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:53:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:53:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:53:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:53:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:53:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:53:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:53:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:53:37.823 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:53:37.816+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a38d753d40 2024-09-22T00:53:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3032 2024-09-22T00:53:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3033 2024-09-22T00:53:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:53:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:53:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:53:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:53:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:53:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:53:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3034 2024-09-22T00:53:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3035 2024-09-22T00:53:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3036 2024-09-22T00:53:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3037 2024-09-22T00:53:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:53:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:53:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:53:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:53:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:53:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:53:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3039 2024-09-22T00:53:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3038 2024-09-22T00:53:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:53:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:53:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:53:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:53:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:53:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:53:41.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3040 2024-09-22T00:53:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3041 2024-09-22T00:53:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:53:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:53:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:53:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:53:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:53:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:53:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3043 2024-09-22T00:53:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3042 2024-09-22T00:53:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:53:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:53:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:53:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:53:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:53:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:53:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3045 2024-09-22T00:53:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3044 2024-09-22T00:53:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:53:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:53:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:53:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:53:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:53:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:53:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:44.075 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:44.076+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281450 2024-09-22T00:53:44.075 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:44.076+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281520 2024-09-22T00:53:44.075 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:44.076+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:53:44.078 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:44.080+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T00:53:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3046 2024-09-22T00:53:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3047 2024-09-22T00:53:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:53:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:53:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:53:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:44.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:44.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:53:44.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:44.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:44.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:53:44.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:53:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3049 2024-09-22T00:53:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3048 2024-09-22T00:53:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:53:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:53:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:53:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:53:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:53:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:53:46.021 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:46.020+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3392ea0 2024-09-22T00:53:46.026 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:46.028+0000 7f54f2d95640 -1 reset not still connected to 0x5617fb4ea4e0 2024-09-22T00:53:46.026 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:46.028+0000 7f54f2d95640 -1 reset not still connected to 0x56180602eb60 2024-09-22T00:53:46.026 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:46.028+0000 7f54f2d95640 -1 reset not still connected to 0x561812608270 2024-09-22T00:53:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3051 2024-09-22T00:53:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3050 2024-09-22T00:53:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:53:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:53:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:53:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:53:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:53:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:53:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3052 2024-09-22T00:53:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:53:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:53:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:53:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3053 2024-09-22T00:53:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:53:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:53:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:53:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3055 2024-09-22T00:53:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:53:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:53:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:53:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3054 2024-09-22T00:53:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:53:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:53:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:53:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:48.843 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:48.844+0000 7f54f2d95640 -1 reset not still connected to 0x5617fb4ea4e0 2024-09-22T00:53:48.847 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:48.844+0000 7f54f2d95640 -1 reset not still connected to 0x56180602eb60 2024-09-22T00:53:48.847 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:48.844+0000 7f54f2d95640 -1 reset not still connected to 0x561812608270 2024-09-22T00:53:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3056 2024-09-22T00:53:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3057 2024-09-22T00:53:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:49.796 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:53:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:53:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:53:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:53:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:53:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:53:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3059 2024-09-22T00:53:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3058 2024-09-22T00:53:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:53:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:53:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:53:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:53:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:53:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:53:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3060 2024-09-22T00:53:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3061 2024-09-22T00:53:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:53:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:53:51.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:53:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:53:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:53:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:53:51.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:53:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:53:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:53:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3062 2024-09-22T00:53:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3063 2024-09-22T00:53:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:53:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:53:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:53:52.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:52.941 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:53:52.940+0000 7f54f2d95640 -1 reset not still connected to 0x5618223aadd0 2024-09-22T00:53:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3064 2024-09-22T00:53:53.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3065 2024-09-22T00:53:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:53:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:53:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:53:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:53:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:53:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:53:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3066 2024-09-22T00:53:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3067 2024-09-22T00:53:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:53:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:53:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:53:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:53:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:53:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:53:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3068 2024-09-22T00:53:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3069 2024-09-22T00:53:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:53:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:53:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:53:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:53:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:53:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:53:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3071 2024-09-22T00:53:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3070 2024-09-22T00:53:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:53:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:53:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:53:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:53:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:53:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:53:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:53:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:53:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:53:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:53:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:53:57.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:53:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3072 2024-09-22T00:53:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3073 2024-09-22T00:53:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3075 2024-09-22T00:53:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:53:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:53:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:53:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3074 2024-09-22T00:53:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:53:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:53:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:53:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3076 2024-09-22T00:53:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3077 2024-09-22T00:53:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:53:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:53:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:53:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:53:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:53:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:53:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:53:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:53:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:53:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:54:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3079 2024-09-22T00:54:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3078 2024-09-22T00:54:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:54:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:54:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:54:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:54:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:54:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:54:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3081 2024-09-22T00:54:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3080 2024-09-22T00:54:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:54:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:54:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:54:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:54:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:54:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:54:01.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:01.946 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:54:01.943+0000 7fab1f496640 -1 reset not still connected to 0x55fa5eb07450 2024-09-22T00:54:01.946 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:54:01.943+0000 7fab1f496640 -1 reset not still connected to 0x55fa5fcfbd40 2024-09-22T00:54:01.959 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:54:01.955+0000 7fab1f496640 -1 reset not still connected to 0x55fa60a0e0d0 2024-09-22T00:54:01.960 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:54:01.955+0000 7fab1f496640 -1 reset not still connected to 0x55fa624f1040 2024-09-22T00:54:01.961 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:54:01.955+0000 7fab1f496640 -1 reset not still connected to 0x55fa665760d0 2024-09-22T00:54:01.961 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:54:01.955+0000 7fab1f496640 -1 reset not still connected to 0x55fa66576410 2024-09-22T00:54:01.961 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:54:01.955+0000 7fab1f496640 -1 reset not still connected to 0x55fa66b985b0 2024-09-22T00:54:01.961 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:54:01.955+0000 7fab1f496640 -1 reset not still connected to 0x55fa6d9a68f0 2024-09-22T00:54:01.961 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:54:01.955+0000 7fab1f496640 -1 reset not still connected to 0x55fa728676c0 2024-09-22T00:54:01.961 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:54:01.955+0000 7fab1f496640 -1 reset not still connected to 0x55fa80ea9040 2024-09-22T00:54:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:54:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:54:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:54:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:54:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:02.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:54:02.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:54:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3083 2024-09-22T00:54:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3082 2024-09-22T00:54:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:03.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3085 2024-09-22T00:54:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3084 2024-09-22T00:54:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:54:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:54:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:54:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:54:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:54:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:54:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3086 2024-09-22T00:54:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3087 2024-09-22T00:54:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:54:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:54:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:54:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:54:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:54:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:54:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3088 2024-09-22T00:54:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3089 2024-09-22T00:54:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:05.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:54:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:54:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:54:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:54:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:54:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:54:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3090 2024-09-22T00:54:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3091 2024-09-22T00:54:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:54:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:54:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:54:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:54:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:54:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:54:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3093 2024-09-22T00:54:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3092 2024-09-22T00:54:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:54:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:54:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:54:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:54:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:54:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:54:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3095 2024-09-22T00:54:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3094 2024-09-22T00:54:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:54:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:54:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:54:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:54:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:54:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:54:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3097 2024-09-22T00:54:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3096 2024-09-22T00:54:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:54:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:54:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:54:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:54:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:54:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:54:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3098 2024-09-22T00:54:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3099 2024-09-22T00:54:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:54:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:54:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:54:10.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:54:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:54:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:54:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3100 2024-09-22T00:54:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3101 2024-09-22T00:54:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:54:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:54:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:54:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:54:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:54:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:54:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3103 2024-09-22T00:54:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3102 2024-09-22T00:54:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:54:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:54:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:54:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3104 2024-09-22T00:54:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3105 2024-09-22T00:54:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:54:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:54:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:54:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:54:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:54:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:54:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3107 2024-09-22T00:54:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3106 2024-09-22T00:54:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:54:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:54:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:54:14.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:14.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:54:14.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:14.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:14.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:54:14.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:54:14.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3108 2024-09-22T00:54:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3109 2024-09-22T00:54:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:54:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:54:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:54:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:54:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:54:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:54:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3110 2024-09-22T00:54:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3111 2024-09-22T00:54:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:54:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:54:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:54:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:54:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:54:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:54:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:54:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:54:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:54:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3112 2024-09-22T00:54:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3113 2024-09-22T00:54:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:54:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:54:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:54:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3114 2024-09-22T00:54:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3115 2024-09-22T00:54:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:54:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:54:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:54:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:54:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:54:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:54:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3117 2024-09-22T00:54:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3116 2024-09-22T00:54:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:54:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:54:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:54:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:54:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:54:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:54:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3119 2024-09-22T00:54:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3118 2024-09-22T00:54:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:54:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:54:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:54:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:54:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:54:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:54:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3121 2024-09-22T00:54:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3120 2024-09-22T00:54:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:54:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:54:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:54:21.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:54:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:54:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:54:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:54:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:54:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:54:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3122 2024-09-22T00:54:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:54:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:54:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:54:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3123 2024-09-22T00:54:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3125 2024-09-22T00:54:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3124 2024-09-22T00:54:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:54:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:54:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:54:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:54:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:54:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:54:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3126 2024-09-22T00:54:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3127 2024-09-22T00:54:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:54:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:54:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:54:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:54:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:54:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:54:25.798 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:54:25.791+0000 7fab1f496640 -1 reset not still connected to 0x55fa66b985b0 2024-09-22T00:54:25.799 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:54:25.791+0000 7fab1f496640 -1 reset not still connected to 0x55fa6d9a68f0 2024-09-22T00:54:25.799 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:54:25.791+0000 7fab1f496640 -1 reset not still connected to 0x55fa728676c0 2024-09-22T00:54:25.800 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:54:25.791+0000 7fab1f496640 -1 reset not still connected to 0x55fa80ea9040 2024-09-22T00:54:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3128 2024-09-22T00:54:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3129 2024-09-22T00:54:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:54:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:54:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:54:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:54:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:54:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:54:25.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3130 2024-09-22T00:54:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3131 2024-09-22T00:54:26.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:54:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:54:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:54:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:54:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:54:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:54:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:54:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:54:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:54:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:54:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:54:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:54:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3132 2024-09-22T00:54:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3133 2024-09-22T00:54:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3135 2024-09-22T00:54:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:54:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:54:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:54:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:54:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:54:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:54:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3137 2024-09-22T00:54:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3136 2024-09-22T00:54:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3134 2024-09-22T00:54:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:29.828 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:54:29.828+0000 7fab5fe19640 -1 reset not still connected to 0x56241da0aa90 2024-09-22T00:54:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:54:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:54:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:54:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:54:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:54:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:54:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3138 2024-09-22T00:54:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3139 2024-09-22T00:54:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:54:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:54:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:54:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:54:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:54:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:54:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3141 2024-09-22T00:54:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3140 2024-09-22T00:54:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:54:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:54:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:54:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:54:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:54:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:54:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:54:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:54:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:54:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:54:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:54:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:54:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3143 2024-09-22T00:54:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3142 2024-09-22T00:54:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3144 2024-09-22T00:54:33.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:33.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:54:33.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:54:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:54:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:54:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:54:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:54:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3147 2024-09-22T00:54:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3145 2024-09-22T00:54:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3146 2024-09-22T00:54:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:54:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:54:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:54:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:54:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:54:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:54:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3149 2024-09-22T00:54:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3148 2024-09-22T00:54:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:54:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:54:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:54:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:54:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:54:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:54:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3150 2024-09-22T00:54:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3151 2024-09-22T00:54:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:54:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:54:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:54:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:54:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:54:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:54:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3152 2024-09-22T00:54:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:54:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:54:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:54:37.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:54:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:54:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:54:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3153 2024-09-22T00:54:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3154 2024-09-22T00:54:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3155 2024-09-22T00:54:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:54:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:54:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:54:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:54:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:54:38.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:54:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:38.836 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3157 2024-09-22T00:54:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3156 2024-09-22T00:54:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:54:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:54:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:54:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:54:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:54:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:54:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3159 2024-09-22T00:54:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3158 2024-09-22T00:54:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:54:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:54:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:54:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3160 2024-09-22T00:54:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3161 2024-09-22T00:54:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:54:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:54:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:54:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:54:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:54:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:54:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:54:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:54:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:54:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3162 2024-09-22T00:54:42.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:54:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:54:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:54:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:54:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:54:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:54:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3164 2024-09-22T00:54:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3165 2024-09-22T00:54:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3163 2024-09-22T00:54:43.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:43.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:54:43.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:43.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:54:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:54:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:54:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:54:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:54:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3166 2024-09-22T00:54:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3167 2024-09-22T00:54:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:54:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:54:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:54:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:54:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:54:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:54:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3168 2024-09-22T00:54:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3169 2024-09-22T00:54:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:54:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:54:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:54:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:45.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:54:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:54:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:54:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3170 2024-09-22T00:54:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3171 2024-09-22T00:54:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:54:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:54:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:54:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:54:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:54:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:54:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3173 2024-09-22T00:54:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:54:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:54:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:54:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:54:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:54:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:54:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3172 2024-09-22T00:54:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3175 2024-09-22T00:54:48.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3174 2024-09-22T00:54:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:54:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:54:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:54:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:54:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:54:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:54:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3177 2024-09-22T00:54:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3176 2024-09-22T00:54:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:54:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:54:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:54:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:54:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:54:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:54:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3179 2024-09-22T00:54:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3178 2024-09-22T00:54:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:54:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:54:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:54:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:54:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:54:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:54:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3181 2024-09-22T00:54:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3180 2024-09-22T00:54:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:54:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:54:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:54:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:51.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:51.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:54:51.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:51.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:54:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:54:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:54:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:54:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:54:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:54:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:54:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:54:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3182 2024-09-22T00:54:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3183 2024-09-22T00:54:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3185 2024-09-22T00:54:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3184 2024-09-22T00:54:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:54:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:54:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:54:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:54:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:54:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:54:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3187 2024-09-22T00:54:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3186 2024-09-22T00:54:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:54:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:54:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:54:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:54:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:54:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:54:55.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3189 2024-09-22T00:54:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3188 2024-09-22T00:54:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:54:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:54:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:54:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:54:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:54:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:54:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:55.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3191 2024-09-22T00:54:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3190 2024-09-22T00:54:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:54:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:54:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:54:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:54:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:54:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:54:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:54:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:54:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:54:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:54:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:54:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:54:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3192 2024-09-22T00:54:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3193 2024-09-22T00:54:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3195 2024-09-22T00:54:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3194 2024-09-22T00:54:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:54:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:54:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:54:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:54:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:54:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:54:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3196 2024-09-22T00:54:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3197 2024-09-22T00:54:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:54:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:54:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:54:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:54:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:54:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:54:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:54:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:54:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:54:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:55:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3199 2024-09-22T00:55:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3198 2024-09-22T00:55:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:00.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:55:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:55:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:55:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:55:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:55:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:55:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3201 2024-09-22T00:55:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3200 2024-09-22T00:55:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:55:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:55:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:55:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:55:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:55:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:55:02.063 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:55:02.063+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3392ea0 2024-09-22T00:55:02.063 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:55:02.063+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3393110 2024-09-22T00:55:02.063 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:55:02.063+0000 7f54f2d95640 -1 reset not still connected to 0x5617f33931e0 2024-09-22T00:55:02.066 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:55:02.067+0000 7f54f2d95640 -1 reset not still connected to 0x5617fb4ea4e0 2024-09-22T00:55:02.066 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:55:02.067+0000 7f54f2d95640 -1 reset not still connected to 0x56180602eb60 2024-09-22T00:55:02.067 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:55:02.067+0000 7f54f2d95640 -1 reset not still connected to 0x561812608270 2024-09-22T00:55:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3202 2024-09-22T00:55:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3203 2024-09-22T00:55:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:55:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:55:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:55:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:55:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:55:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:55:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3205 2024-09-22T00:55:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3204 2024-09-22T00:55:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:55:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:55:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:55:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:55:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:55:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:55:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3206 2024-09-22T00:55:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3207 2024-09-22T00:55:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:55:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:55:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:55:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:55:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:55:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:55:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3209 2024-09-22T00:55:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3208 2024-09-22T00:55:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:55:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:55:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:55:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:55:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:55:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:55:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3210 2024-09-22T00:55:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:55:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:55:06.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:55:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3211 2024-09-22T00:55:06.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:06.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:55:06.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:06.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:55:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:55:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3212 2024-09-22T00:55:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:55:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:55:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:55:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3213 2024-09-22T00:55:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:55:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:55:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:55:07.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3214 2024-09-22T00:55:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3215 2024-09-22T00:55:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:55:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:55:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:55:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:55:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:55:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:55:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3217 2024-09-22T00:55:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3216 2024-09-22T00:55:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:55:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:55:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:55:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:55:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:55:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:55:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3218 2024-09-22T00:55:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3219 2024-09-22T00:55:10.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:10.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:55:10.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:10.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:10.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:55:10.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:55:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:55:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:55:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:55:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:10.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3221 2024-09-22T00:55:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3220 2024-09-22T00:55:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:55:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:55:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:55:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:55:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:55:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:55:11.901 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:55:11.899+0000 7fab5fe19640 -1 reset not still connected to 0x56241da0aa90 2024-09-22T00:55:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3223 2024-09-22T00:55:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:55:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:55:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:55:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:55:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:55:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:55:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3222 2024-09-22T00:55:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3225 2024-09-22T00:55:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3224 2024-09-22T00:55:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:55:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:55:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:55:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:55:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:55:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:55:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3227 2024-09-22T00:55:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3226 2024-09-22T00:55:14.795 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:55:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:55:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:55:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:55:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:55:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:55:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3229 2024-09-22T00:55:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3228 2024-09-22T00:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:55:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:55:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:15.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3230 2024-09-22T00:55:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3231 2024-09-22T00:55:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:55:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:55:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:55:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:55:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:55:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:55:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:55:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:55:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:55:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:55:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:55:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:55:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3233 2024-09-22T00:55:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3232 2024-09-22T00:55:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3234 2024-09-22T00:55:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3235 2024-09-22T00:55:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:55:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:55:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:55:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:55:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:55:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:55:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3237 2024-09-22T00:55:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3236 2024-09-22T00:55:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:55:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:55:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:55:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:55:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:55:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:55:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3239 2024-09-22T00:55:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3238 2024-09-22T00:55:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:55:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:55:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:55:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:55:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:55:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:55:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3241 2024-09-22T00:55:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3240 2024-09-22T00:55:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:55:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:55:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:55:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:55:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:55:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:55:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:55:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:55:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:55:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:55:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:55:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:55:22.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3243 2024-09-22T00:55:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3242 2024-09-22T00:55:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3244 2024-09-22T00:55:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:55:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:55:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:55:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3245 2024-09-22T00:55:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:55:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:55:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:55:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3246 2024-09-22T00:55:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3247 2024-09-22T00:55:24.783 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:55:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:55:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:55:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:55:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:55:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:55:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3249 2024-09-22T00:55:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3248 2024-09-22T00:55:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:25.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:55:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:55:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:55:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:55:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:55:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:55:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3250 2024-09-22T00:55:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3251 2024-09-22T00:55:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:55:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:55:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:55:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:55:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:55:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:55:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:55:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:55:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:55:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:55:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:55:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:55:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3252 2024-09-22T00:55:27.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3253 2024-09-22T00:55:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:55:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:55:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:55:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:55:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:55:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:55:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3255 2024-09-22T00:55:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3254 2024-09-22T00:55:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3256 2024-09-22T00:55:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3257 2024-09-22T00:55:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:29.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:29.977 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:55:29.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:29.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:29.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:55:29.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:55:29.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:55:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:55:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:55:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3259 2024-09-22T00:55:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3258 2024-09-22T00:55:30.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:55:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:55:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:55:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:55:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:55:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:55:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3260 2024-09-22T00:55:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3261 2024-09-22T00:55:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:55:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:55:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:55:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:55:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:55:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:55:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3262 2024-09-22T00:55:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3263 2024-09-22T00:55:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:55:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:55:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:55:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:55:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:55:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:55:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:32.859 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:55:32.851+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a3630960d0 2024-09-22T00:55:32.860 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:55:32.855+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a38d753d40 2024-09-22T00:55:32.890 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:55:32.891+0000 7f54f2d95640 -1 reset not still connected to 0x5617f706be10 2024-09-22T00:55:32.891 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:55:32.891+0000 7f54f2d95640 -1 reset not still connected to 0x5618223aadd0 2024-09-22T00:55:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3264 2024-09-22T00:55:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3265 2024-09-22T00:55:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:55:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:55:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:55:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:55:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:55:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:55:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:33.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3266 2024-09-22T00:55:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3267 2024-09-22T00:55:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:34.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:34.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:55:34.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:34.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:34.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:55:34.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:55:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:55:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:55:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:55:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:34.978 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2024-09-22T00:55:34.979+0000 7f1bcc00c640 -1 reset not still connected to 0x55f7e424a4e0 2024-09-22T00:55:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3268 2024-09-22T00:55:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3269 2024-09-22T00:55:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:55:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:55:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:55:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:55:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:55:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:55:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3271 2024-09-22T00:55:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3270 2024-09-22T00:55:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:55:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:55:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:55:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:55:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:55:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:55:37.033 INFO:tasks.ceph.cluster2.osd.1.smithi060.stderr:2024-09-22T00:55:37.031+0000 7fc7b6c4b640 -1 reset not still connected to 0x55a38d753d40 2024-09-22T00:55:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3273 2024-09-22T00:55:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3272 2024-09-22T00:55:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:55:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:55:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:55:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:55:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:55:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:55:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:37.989 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:55:37.979+0000 7fab1f496640 -1 reset not still connected to 0x55fa80ea9040 2024-09-22T00:55:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3275 2024-09-22T00:55:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3274 2024-09-22T00:55:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:55:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:55:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:55:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:55:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:55:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:55:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3276 2024-09-22T00:55:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3277 2024-09-22T00:55:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:55:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:55:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:55:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:55:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:55:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:55:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3278 2024-09-22T00:55:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3279 2024-09-22T00:55:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:55:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:55:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:55:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:55:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:55:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:55:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3280 2024-09-22T00:55:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3281 2024-09-22T00:55:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:55:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:55:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:55:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:55:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:55:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:55:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:55:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:55:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:55:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:55:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:55:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:55:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3283 2024-09-22T00:55:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3282 2024-09-22T00:55:42.917 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3284 2024-09-22T00:55:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3285 2024-09-22T00:55:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:55:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:55:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:55:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:55:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:55:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:55:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3287 2024-09-22T00:55:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3286 2024-09-22T00:55:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:55:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:55:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:55:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-22T00:55:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:55:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-22T00:55:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3288 2024-09-22T00:55:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3289 2024-09-22T00:55:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-22T00:55:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:55:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-22T00:55:45.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:45.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:55:45.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:45.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:45.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:55:45.939 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:55:45.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3290 2024-09-22T00:55:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3291 2024-09-22T00:55:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-22T00:55:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:55:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-22T00:55:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:55:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:55:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:55:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:55:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:55:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:55:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-22T00:55:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:55:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-22T00:55:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3293 2024-09-22T00:55:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3292 2024-09-22T00:55:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3295 2024-09-22T00:55:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3294 2024-09-22T00:55:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-22T00:55:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:55:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-22T00:55:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:55:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:55:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:55:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3296 2024-09-22T00:55:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3297 2024-09-22T00:55:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-22T00:55:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:55:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-22T00:55:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:55:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 131683 2024-09-22T00:55:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3299 2024-09-22T00:55:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3298 2024-09-22T00:55:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-22T00:55:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:55:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-22T00:55:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:55:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 131815 2024-09-22T00:55:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3300 2024-09-22T00:55:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-22T00:55:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:55:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-22T00:55:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:55:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 131942 2024-09-22T00:55:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-22T00:55:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:55:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-22T00:55:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:55:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 132089 2024-09-22T00:55:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3301 2024-09-22T00:55:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3302 2024-09-22T00:55:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for i in {1..30} 2024-09-22T00:55:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-22T00:55:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:55:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:55:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:55:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-22T00:55:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:55:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 132236 2024-09-22T00:55:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3303 2024-09-22T00:55:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:54.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:55:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 132384 2024-09-22T00:55:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Snapshot ID: 3304 2024-09-22T00:55:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 3 2024-09-22T00:55:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:55:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 132532 2024-09-22T00:55:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:55:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 132680 2024-09-22T00:55:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-22T00:55:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 132848 2024-09-22T00:55:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:55:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:55:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 131562 2024-09-22T00:56:10.532 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:56:10.530+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T00:56:10.538 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T00:56:10.538+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T00:56:50.520 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2024-09-22T00:56:50.517+0000 7fab5fe19640 -1 reset not still connected to 0x5623f97bd860 2024-09-22T00:58:22.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:58:22.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:58:22.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:58:22.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 131684 2024-09-22T00:58:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:58:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:58:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:58:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 131816 2024-09-22T00:58:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:58:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:58:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:58:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 131943 2024-09-22T00:58:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:58:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:58:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:58:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 132090 2024-09-22T00:58:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:58:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:58:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:58:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 132237 2024-09-22T00:58:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:58:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:58:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:58:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 132385 2024-09-22T00:58:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:58:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:58:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:58:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 132533 2024-09-22T00:58:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:58:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:58:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:58:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 132681 2024-09-22T00:58:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:58:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:58:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-22T00:58:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 132849 2024-09-22T00:58:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ret=124 2024-09-22T00:58:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( ret != 124 )) 2024-09-22T00:58:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:58:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:58:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:58:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 1 2024-09-22T00:58:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:58:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd0 2024-09-22T00:58:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary1 2024-09-22T00:58:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary1 2024-09-22T00:58:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:58:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary1 2024-09-22T00:58:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:58:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:58:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 2 2024-09-22T00:58:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd1 2024-09-22T00:58:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary2 2024-09-22T00:58:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary2 2024-09-22T00:58:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:58:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:58:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary2 2024-09-22T00:58:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:58:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 3 2024-09-22T00:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd2 2024-09-22T00:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary3 2024-09-22T00:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary3 2024-09-22T00:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 4 2024-09-22T00:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary3 2024-09-22T00:58:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:58:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:58:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd3 2024-09-22T00:58:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary4 2024-09-22T00:58:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary4 2024-09-22T00:58:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 5 2024-09-22T00:58:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:58:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary4 2024-09-22T00:58:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:58:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd4 2024-09-22T00:58:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary5 2024-09-22T00:58:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary5 2024-09-22T00:58:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:58:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:58:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary5 2024-09-22T00:58:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 6 2024-09-22T00:58:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd5 2024-09-22T00:58:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary6 2024-09-22T00:58:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:58:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary6 2024-09-22T00:58:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:58:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary6 2024-09-22T00:58:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:58:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:58:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:58:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:58:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:58:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ COMPARE_PIDS+=($!) 2024-09-22T00:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 139024 2024-09-22T00:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 9 2024-09-22T00:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd8 2024-09-22T00:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary9 2024-09-22T00:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary9 2024-09-22T00:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary9 2024-09-22T00:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 8 2024-09-22T00:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd7 2024-09-22T00:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary8 2024-09-22T00:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary8 2024-09-22T00:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary8 2024-09-22T00:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 7 2024-09-22T00:58:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd6 2024-09-22T00:58:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary7 2024-09-22T00:58:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary7 2024-09-22T00:58:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:58:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary7 2024-09-22T00:58:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ compare_demoted_promoted_image 10 2024-09-22T00:58:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local dev=/dev/rbd9 2024-09-22T00:58:30.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local img=image-primary10 2024-09-22T00:58:30.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local mntpt=test-primary10 2024-09-22T00:58:30.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local demote_md5 promote_md5 2024-09-22T00:58:30.510 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo umount test-primary10 2024-09-22T00:58:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-22T00:58:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:58:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-22T00:58:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:58:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-22T00:58:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:58:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-22T00:58:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:58:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-22T00:58:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:58:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-22T00:58:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:58:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-22T00:58:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:58:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-22T00:58:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:58:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-22T00:58:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-22T00:58:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:58:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:59:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=ff9c8732bbba5de009ea0ab831f19e34 2024-09-22T00:59:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-22T00:59:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-22T00:59:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:59:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-22T00:59:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=cb879ab53bf9043914f34a512ea65640 2024-09-22T00:59:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-22T00:59:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-22T00:59:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:59:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-22T00:59:03.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=f3e2268f9828467c5e9fcdc294c7a95e 2024-09-22T00:59:03.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-22T00:59:03.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-22T00:59:03.670 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:59:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-22T00:59:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=fcaa39fe33ca545582bd26df57acf268 2024-09-22T00:59:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-22T00:59:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=31d0703d5459b67ba70a20cd448c40d2 2024-09-22T00:59:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-22T00:59:03.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=5a60ac29f4bd6e31e0cda7ee082d7f67 2024-09-22T00:59:03.834 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-22T00:59:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-22T00:59:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:59:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-22T00:59:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=b8bcede3c6796fe711c30ba3a5d6a149 2024-09-22T00:59:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-22T00:59:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-22T00:59:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:59:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-22T00:59:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-22T00:59:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:59:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-22T00:59:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-22T00:59:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:59:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-22T00:59:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=1bd94c4db1937a32ed4214be51519f4c 2024-09-22T00:59:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-22T00:59:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-22T00:59:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:59:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-22T00:59:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=5ee574af460f345e9e6b13547cefde56 2024-09-22T00:59:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-22T00:59:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-22T00:59:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:59:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-22T00:59:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_md5=08cdf9bbd5d4ba3084809cefa26fb9b1 2024-09-22T00:59:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-22T00:59:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-22T00:59:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:59:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-22T00:59:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:59:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:59:05.454 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:59:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-22T00:59:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:59:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-22T00:59:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:59:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-22T00:59:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:59:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:59:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-22T00:59:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:59:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:59:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:59:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:59:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:59:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-22T00:59:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:59:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-22T00:59:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:59:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-22T00:59:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:59:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-22T00:59:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:59:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:59:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image demoted to non-primary 2024-09-22T00:59:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-22T00:59:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-22T00:59:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:59:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:59:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-22T00:59:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:59:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T00:59:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-22T00:59:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-22T00:59:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:59:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:59:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T00:59:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T00:59:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-22T00:59:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:59:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:59:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:59:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 4de603dc-70a4-4d8e-ac7a-d2f59bde04b9 2024-09-22T00:59:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05 2024-09-22T00:59:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:59:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:59:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:06' 2024-09-22T00:59:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:59:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8102f56b-719c-4239-9d1a-177deba63ef3 2024-09-22T00:59:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05 2024-09-22T00:59:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:59:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:59:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2762752.0,"last_snapshot_bytes":143360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966553,"remote_snapshot_timestamp":1726966553,"replay_state":"idle"} 2024-09-22T00:59:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05' 2024-09-22T00:59:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:59:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:59:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:06.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:06.643 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:06.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:06.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-22 00:59:05' 2024-09-22T00:59:06.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:06.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:59:06.644 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:05' 2024-09-22T00:59:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-22 00:59:05' 2024-09-22T00:59:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:59:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:05' 2024-09-22T00:59:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:59:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: d8084146-8956-435b-a81f-43c80a014180 2024-09-22T00:59:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05 2024-09-22T00:59:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:59:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:59:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3512320.0,"last_snapshot_bytes":1933312,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966548,"remote_snapshot_timestamp":1726966548,"replay_state":"idle"} 2024-09-22T00:59:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05' 2024-09-22T00:59:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:59:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-22 00:59:05' 2024-09-22T00:59:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:59:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:05' 2024-09-22T00:59:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-22T00:59:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:59:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T00:59:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-22T00:59:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:59:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-22T00:59:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:59:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:59:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:59:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-22T00:59:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:59:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:59:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:59:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 0f6d12b8-9468-4767-b8f3-f62a5392a241 2024-09-22T00:59:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05 2024-09-22T00:59:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:59:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:59:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:06' 2024-09-22T00:59:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:59:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-22 00:59:05' 2024-09-22T00:59:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:59:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:05' 2024-09-22T00:59:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:59:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 86d5828f-ec5b-4c64-81bd-fc524d8f94cf 2024-09-22T00:59:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05 2024-09-22T00:59:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:59:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:59:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4603904.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966554,"remote_snapshot_timestamp":1726966554,"replay_state":"idle"} 2024-09-22T00:59:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05' 2024-09-22T00:59:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:59:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-22 00:59:05' 2024-09-22T00:59:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:59:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:05' 2024-09-22T00:59:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4603904.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966554,"remote_snapshot_timestamp":1726966554,"replay_state":"idle"}2024-09-22 00:59:05image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966546,"remote_snapshot_timestamp":1726966546,"replay_state":"idle"}2024-09-22 00:59:05image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3581952.0,"last_snapshot_bytes":7020544,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966547,"remote_snapshot_timestamp":1726966547,"replay_state":"idle"}2024-09-22 00:59:05image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3512320.0,"last_snapshot_bytes":1933312,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966548,"remote_snapshot_timestamp":1726966548,"replay_state":"idle"}2024-09-22 00:59:05image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1966080.0,"last_snapshot_bytes":3772416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966549,"remote_snapshot_timestamp":1726966549,"replay_state":"idle"}2024-09-22 00:59:05image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1703936.0,"last_snapshot_bytes":1667072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966551,"remote_snapshot_timestamp":1726966551,"replay_state":"idle"}2024-09-22 00:59:05image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2762752.0,"last_snapshot_bytes":143360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966553,"remote_snapshot_timestamp":1726966553,"replay_state":"idle"}2024-09-22 00:59:05' 2024-09-22T00:59:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:59:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:59:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 199f939a-c91f-4933-85bd-fed08290fd6e 2024-09-22T00:59:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05 2024-09-22T00:59:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:59:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:59:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1966080.0,"last_snapshot_bytes":3772416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966549,"remote_snapshot_timestamp":1726966549,"replay_state":"idle"} 2024-09-22T00:59:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05' 2024-09-22T00:59:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4603904.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966554,"remote_snapshot_timestamp":1726966554,"replay_state":"idle"}2024-09-22 00:59:05image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966546,"remote_snapshot_timestamp":1726966546,"replay_state":"idle"}2024-09-22 00:59:05image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3581952.0,"last_snapshot_bytes":7020544,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966547,"remote_snapshot_timestamp":1726966547,"replay_state":"idle"}2024-09-22 00:59:05image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3512320.0,"last_snapshot_bytes":1933312,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966548,"remote_snapshot_timestamp":1726966548,"replay_state":"idle"}2024-09-22 00:59:05image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1966080.0,"last_snapshot_bytes":3772416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966549,"remote_snapshot_timestamp":1726966549,"replay_state":"idle"}2024-09-22 00:59:05image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1703936.0,"last_snapshot_bytes":1667072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966551,"remote_snapshot_timestamp":1726966551,"replay_state":"idle"}2024-09-22 00:59:05image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2762752.0,"last_snapshot_bytes":143360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966553,"remote_snapshot_timestamp":1726966553,"replay_state":"idle"}2024-09-22 00:59:05' 2024-09-22T00:59:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:59:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:59:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:59:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: c7a13db2-2e62-40a8-86e3-e19844d44822 2024-09-22T00:59:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05 2024-09-22T00:59:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:59:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:59:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966546,"remote_snapshot_timestamp":1726966546,"replay_state":"idle"} 2024-09-22T00:59:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05' 2024-09-22T00:59:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:59:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-22 00:59:05' 2024-09-22T00:59:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:59:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:05' 2024-09-22T00:59:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4603904.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966554,"remote_snapshot_timestamp":1726966554,"replay_state":"idle"}2024-09-22 00:59:05image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966546,"remote_snapshot_timestamp":1726966546,"replay_state":"idle"}2024-09-22 00:59:05image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3581952.0,"last_snapshot_bytes":7020544,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966547,"remote_snapshot_timestamp":1726966547,"replay_state":"idle"}2024-09-22 00:59:05image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3512320.0,"last_snapshot_bytes":1933312,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966548,"remote_snapshot_timestamp":1726966548,"replay_state":"idle"}2024-09-22 00:59:05image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1966080.0,"last_snapshot_bytes":3772416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966549,"remote_snapshot_timestamp":1726966549,"replay_state":"idle"}2024-09-22 00:59:05image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1703936.0,"last_snapshot_bytes":1667072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966551,"remote_snapshot_timestamp":1726966551,"replay_state":"idle"}2024-09-22 00:59:05image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2762752.0,"last_snapshot_bytes":143360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966553,"remote_snapshot_timestamp":1726966553,"replay_state":"idle"}2024-09-22 00:59:05' 2024-09-22T00:59:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:59:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-22 00:59:05' 2024-09-22T00:59:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:59:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:05' 2024-09-22T00:59:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:59:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 078a49ba-3e14-42c8-acc1-269d8a4f54d9 2024-09-22T00:59:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05 2024-09-22T00:59:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:59:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:59:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:06' 2024-09-22T00:59:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:59:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:06.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:06.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:06.892 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-22 00:59:05' 2024-09-22T00:59:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:59:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:05' 2024-09-22T00:59:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:59:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:59:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:59:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-22T00:59:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-22T00:59:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:59:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:59:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-22T00:59:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:59:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4603904.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966554,"remote_snapshot_timestamp":1726966554,"replay_state":"idle"}2024-09-22 00:59:05image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966546,"remote_snapshot_timestamp":1726966546,"replay_state":"idle"}2024-09-22 00:59:05image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3581952.0,"last_snapshot_bytes":7020544,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966547,"remote_snapshot_timestamp":1726966547,"replay_state":"idle"}2024-09-22 00:59:05image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3512320.0,"last_snapshot_bytes":1933312,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966548,"remote_snapshot_timestamp":1726966548,"replay_state":"idle"}2024-09-22 00:59:05image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1966080.0,"last_snapshot_bytes":3772416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966549,"remote_snapshot_timestamp":1726966549,"replay_state":"idle"}2024-09-22 00:59:05image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1703936.0,"last_snapshot_bytes":1667072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966551,"remote_snapshot_timestamp":1726966551,"replay_state":"idle"}2024-09-22 00:59:05image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2762752.0,"last_snapshot_bytes":143360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966553,"remote_snapshot_timestamp":1726966553,"replay_state":"idle"}2024-09-22 00:59:05' 2024-09-22T00:59:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:59:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:59:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-22T00:59:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:59:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4603904.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966554,"remote_snapshot_timestamp":1726966554,"replay_state":"idle"}2024-09-22 00:59:05image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966546,"remote_snapshot_timestamp":1726966546,"replay_state":"idle"}2024-09-22 00:59:05image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3581952.0,"last_snapshot_bytes":7020544,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966547,"remote_snapshot_timestamp":1726966547,"replay_state":"idle"}2024-09-22 00:59:05image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3512320.0,"last_snapshot_bytes":1933312,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966548,"remote_snapshot_timestamp":1726966548,"replay_state":"idle"}2024-09-22 00:59:05image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1966080.0,"last_snapshot_bytes":3772416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966549,"remote_snapshot_timestamp":1726966549,"replay_state":"idle"}2024-09-22 00:59:05image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1703936.0,"last_snapshot_bytes":1667072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966551,"remote_snapshot_timestamp":1726966551,"replay_state":"idle"}2024-09-22 00:59:05image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2762752.0,"last_snapshot_bytes":143360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966553,"remote_snapshot_timestamp":1726966553,"replay_state":"idle"}2024-09-22 00:59:05' 2024-09-22T00:59:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:59:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:59:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:07.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:07.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-22T00:59:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:59:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4603904.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966554,"remote_snapshot_timestamp":1726966554,"replay_state":"idle"}2024-09-22 00:59:05image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stopping_replayremote image demoted2024-09-22 00:59:07image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3581952.0,"last_snapshot_bytes":7020544,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966547,"remote_snapshot_timestamp":1726966547,"replay_state":"idle"}2024-09-22 00:59:05image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stopping_replayremote image demoted2024-09-22 00:59:07image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1966080.0,"last_snapshot_bytes":3772416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966549,"remote_snapshot_timestamp":1726966549,"replay_state":"idle"}2024-09-22 00:59:05image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stopping_replayremote image demoted2024-09-22 00:59:07image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stopping_replayremote image demoted2024-09-22 00:59:07' 2024-09-22T00:59:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:59:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stopping_replayremote image demoted2024-09-22 00:59:07image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stopping_replayremote image demoted2024-09-22 00:59:07image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3581952.0,"last_snapshot_bytes":7020544,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966547,"remote_snapshot_timestamp":1726966547,"replay_state":"idle"}2024-09-22 00:59:05image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:07image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1966080.0,"last_snapshot_bytes":3772416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966549,"remote_snapshot_timestamp":1726966549,"replay_state":"idle"}2024-09-22 00:59:05image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stopping_replayremote image demoted2024-09-22 00:59:07image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2762752.0,"last_snapshot_bytes":143360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966553,"remote_snapshot_timestamp":1726966553,"replay_state":"idle"}2024-09-22 00:59:05' 2024-09-22T00:59:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:59:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:59:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:07.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:59:07.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stopping_replayremote image demoted2024-09-22 00:59:07image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stopping_replayremote image demoted2024-09-22 00:59:07image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stopping_replayremote image demoted2024-09-22 00:59:07image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stopping_replayremote image demoted2024-09-22 00:59:07image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stopping_replayremote image demoted2024-09-22 00:59:07image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stopping_replayremote image demoted2024-09-22 00:59:07image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stopping_replayremote image demoted2024-09-22 00:59:07' 2024-09-22T00:59:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:59:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-22T00:59:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:59:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-22T00:59:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:59:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:07.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:07.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:07.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:59:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-22T00:59:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:59:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-22T00:59:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:59:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T00:59:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-22T00:59:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:59:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:59:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:59:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: ec869a35-b7df-4a94-a692-4cf4ac04e207 2024-09-22T00:59:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05 2024-09-22T00:59:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:59:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:59:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:07' 2024-09-22T00:59:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:59:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-22 00:59:05' 2024-09-22T00:59:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:59:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:05' 2024-09-22T00:59:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:59:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6175a8d9-0e7f-4d09-9b55-cc3a19b9272f 2024-09-22T00:59:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05 2024-09-22T00:59:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:59:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:59:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:07' 2024-09-22T00:59:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:07.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image is not primary 2024-09-22T00:59:07.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:07.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:07.591 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-22 00:59:05' 2024-09-22T00:59:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:59:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:05' 2024-09-22T00:59:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:07image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:07image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:07image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:07image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:07image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:07image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:07' 2024-09-22T00:59:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:59:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:59:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-22T00:59:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:59:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:07.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:07image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:07image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:07image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:07image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:07image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:07image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:06image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image is not primary42614264mirror.0smithi0602024-09-22 00:59:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+unknownremote image demoted2024-09-22 00:59:07' 2024-09-22T00:59:07.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:07.753 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:59:07.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:05' 2024-09-22T00:59:07.757 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:59:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-22T00:59:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:59:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-22T00:59:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-22T00:59:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:59:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-22T00:59:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:59:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:59:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-22T00:59:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-22T00:59:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-22T00:59:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-22T00:59:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:59:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-22T00:59:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T00:59:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 4de603dc-70a4-4d8e-ac7a-d2f59bde04b9 2024-09-22T00:59:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:59:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:06 2024-09-22T00:59:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:59:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05' 2024-09-22T00:59:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T00:59:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8102f56b-719c-4239-9d1a-177deba63ef3 2024-09-22T00:59:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:59:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:07 2024-09-22T00:59:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:59:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05' 2024-09-22T00:59:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:59:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: d8084146-8956-435b-a81f-43c80a014180 2024-09-22T00:59:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:59:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:07 2024-09-22T00:59:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:59:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05' 2024-09-22T00:59:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:59:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:59:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:59:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:07' 2024-09-22T00:59:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-22 00:59:07' 2024-09-22T00:59:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:59:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:07' 2024-09-22T00:59:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:06' 2024-09-22T00:59:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-22 00:59:06' 2024-09-22T00:59:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:59:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:07' 2024-09-22T00:59:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:06' 2024-09-22T00:59:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-22 00:59:07' 2024-09-22T00:59:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:59:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:07' 2024-09-22T00:59:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:59:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 0f6d12b8-9468-4767-b8f3-f62a5392a241 2024-09-22T00:59:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:59:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:06 2024-09-22T00:59:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:59:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05' 2024-09-22T00:59:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:59:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:06' 2024-09-22T00:59:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-22 00:59:06' 2024-09-22T00:59:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:59:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:06' 2024-09-22T00:59:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-22T00:59:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:59:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-22T00:59:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-22T00:59:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:59:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-22T00:59:08.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-22T00:59:08.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:59:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-22T00:59:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-22T00:59:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:59:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-22T00:59:08.141 INFO:tasks.ceph.cluster2.osd.0.smithi060.stderr:2024-09-22T00:59:08.137+0000 7fab1f496640 -1 reset not still connected to 0x55fa63439ba0 2024-09-22T00:59:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T00:59:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 078a49ba-3e14-42c8-acc1-269d8a4f54d9 2024-09-22T00:59:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:59:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:06 2024-09-22T00:59:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:59:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05' 2024-09-22T00:59:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:59:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:06' 2024-09-22T00:59:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-22 00:59:06' 2024-09-22T00:59:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:59:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:06' 2024-09-22T00:59:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05' 2024-09-22T00:59:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:59:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:07' 2024-09-22T00:59:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:59:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-22T00:59:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:59:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:59:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-22T00:59:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05' 2024-09-22T00:59:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T00:59:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:06' 2024-09-22T00:59:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T00:59:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T00:59:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 86d5828f-ec5b-4c64-81bd-fc524d8f94cf 2024-09-22T00:59:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:59:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:07 2024-09-22T00:59:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:59:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05' 2024-09-22T00:59:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05' 2024-09-22T00:59:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T00:59:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:07' 2024-09-22T00:59:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T00:59:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:59:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 199f939a-c91f-4933-85bd-fed08290fd6e 2024-09-22T00:59:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:59:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:07 2024-09-22T00:59:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:59:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05' 2024-09-22T00:59:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:59:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:59:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-22T00:59:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T00:59:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:59:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-22T00:59:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:07' 2024-09-22T00:59:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-22 00:59:07' 2024-09-22T00:59:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:59:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:07' 2024-09-22T00:59:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841+ status='image-primary2: 2024-09-22T00:59:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: c7a13db2-2e62-40a8-86e3-e19844d44822 2024-09-22T00:59:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:59:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:07 2024-09-22T00:59:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:59:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05' 2024-09-22T00:59:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05' 2024-09-22T00:59:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:59:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-22T00:59:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T00:59:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:59:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-22T00:59:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:59:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:07' 2024-09-22T00:59:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-22 00:59:07' 2024-09-22T00:59:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:59:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:07' 2024-09-22T00:59:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:06' 2024-09-22T00:59:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:59:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-22T00:59:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:59:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:59:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-22T00:59:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:07' 2024-09-22T00:59:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-22 00:59:07' 2024-09-22T00:59:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:59:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:07' 2024-09-22T00:59:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05' 2024-09-22T00:59:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T00:59:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:06' 2024-09-22T00:59:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T00:59:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:08.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.432 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-22T00:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T00:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:59:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-22T00:59:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05' 2024-09-22T00:59:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:59:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:07' 2024-09-22T00:59:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:59:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05' 2024-09-22T00:59:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T00:59:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-22T00:59:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:59:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:59:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-22T00:59:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:07' 2024-09-22T00:59:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T00:59:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-22T00:59:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T00:59:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:59:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-22T00:59:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05' 2024-09-22T00:59:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:59:08.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:07' 2024-09-22T00:59:08.584 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:59:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-22T00:59:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:59:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:59:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-22T00:59:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-22T00:59:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:59:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-22T00:59:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-22T00:59:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:59:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-22T00:59:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T00:59:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: ec869a35-b7df-4a94-a692-4cf4ac04e207 2024-09-22T00:59:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:59:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:07 2024-09-22T00:59:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:59:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05' 2024-09-22T00:59:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:59:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:07' 2024-09-22T00:59:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-22 00:59:07' 2024-09-22T00:59:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:59:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:07' 2024-09-22T00:59:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:59:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6175a8d9-0e7f-4d09-9b55-cc3a19b9272f 2024-09-22T00:59:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image demoted 2024-09-22T00:59:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:07 2024-09-22T00:59:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster1 2024-09-22T00:59:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+unknown 2024-09-22T00:59:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: remote image is not primary 2024-09-22T00:59:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:05' 2024-09-22T00:59:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+unknown' 2024-09-22T00:59:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+unknown 2024-09-22T00:59:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: remote image demoted 2024-09-22T00:59:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:07' 2024-09-22T00:59:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-22 00:59:07' 2024-09-22T00:59:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:59:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+unknown 2024-09-22T00:59:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:07' 2024-09-22T00:59:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05' 2024-09-22T00:59:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T00:59:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:07' 2024-09-22T00:59:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T00:59:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-22T00:59:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T00:59:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:59:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-22T00:59:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='WARNINGOKWARNING1042634269mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary4d8084146-8956-435b-a81f-43c80a014180up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:06cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+unknownremote image demoted42634269mirror.0smithi0602024-09-22 00:59:07cluster16e00ee5e-7588-41be-ae6d-d48841ed4326up+unknownremote image is not primary2024-09-22 00:59:05' 2024-09-22T00:59:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:59:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:07' 2024-09-22T00:59:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:59:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+unknown 2024-09-22T00:59:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+unknown 2024-09-22T00:59:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+unknown 2024-09-22T00:59:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+unknown 2024-09-22T00:59:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-22T00:59:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:59:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local force= 2024-09-22T00:59:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-22T00:59:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:59:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:59:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:59:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:59:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:59:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:59:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:59:09.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:59:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:59:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:59:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-22T00:59:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:59:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:59:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-22T00:59:09.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:59:09.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:59:09.079 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-22T00:59:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:59:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:59:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-22T00:59:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:59:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:59:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:59:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:59:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-22T00:59:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-22T00:59:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:59:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:59:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:59:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:59:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-22T00:59:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-22T00:59:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd1 2024-09-22T00:59:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd1 2024-09-22T00:59:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:59:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd0 2024-09-22T00:59:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd0 2024-09-22T00:59:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:59:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd2 2024-09-22T00:59:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:59:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd2 2024-09-22T00:59:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd6 2024-09-22T00:59:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd6 2024-09-22T00:59:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd4 2024-09-22T00:59:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd4 2024-09-22T00:59:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:59:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:59:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd5 2024-09-22T00:59:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd5 2024-09-22T00:59:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:59:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd7 2024-09-22T00:59:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd3 2024-09-22T00:59:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd7 2024-09-22T00:59:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd3 2024-09-22T00:59:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:59:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:59:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:59:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:Image promoted to primary 2024-09-22T00:59:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:59:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:59:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \n\b\d ]] 2024-09-22T00:59:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-22T00:59:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-22T00:59:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-22T00:59:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd8 2024-09-22T00:59:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:59:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd8 2024-09-22T00:59:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ dev=/dev/rbd9 2024-09-22T00:59:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sudo md5sum /dev/rbd9 2024-09-22T00:59:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ awk '{print $1}' 2024-09-22T00:59:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=f3e2268f9828467c5e9fcdc294c7a95e 2024-09-22T00:59:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-22T00:59:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ f3e2268f9828467c5e9fcdc294c7a95e != \f\3\e\2\2\6\8\f\9\8\2\8\4\6\7\c\5\e\9\f\c\d\c\2\9\4\c\7\a\9\5\e ]] 2024-09-22T00:59:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=31d0703d5459b67ba70a20cd448c40d2 2024-09-22T00:59:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-22T00:59:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 31d0703d5459b67ba70a20cd448c40d2 != \3\1\d\0\7\0\3\d\5\4\5\9\b\6\7\b\a\7\0\a\2\0\c\d\4\4\8\c\4\0\d\2 ]] 2024-09-22T00:59:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=b8bcede3c6796fe711c30ba3a5d6a149 2024-09-22T00:59:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-22T00:59:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=5a60ac29f4bd6e31e0cda7ee082d7f67 2024-09-22T00:59:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-22T00:59:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ b8bcede3c6796fe711c30ba3a5d6a149 != \b\8\b\c\e\d\e\3\c\6\7\9\6\f\e\7\1\1\c\3\0\b\a\3\a\5\d\6\a\1\4\9 ]] 2024-09-22T00:59:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=1bd94c4db1937a32ed4214be51519f4c 2024-09-22T00:59:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-22T00:59:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=cb879ab53bf9043914f34a512ea65640 2024-09-22T00:59:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-22T00:59:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 5a60ac29f4bd6e31e0cda7ee082d7f67 != \5\a\6\0\a\c\2\9\f\4\b\d\6\e\3\1\e\0\c\d\a\7\e\e\0\8\2\d\7\f\6\7 ]] 2024-09-22T00:59:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 1bd94c4db1937a32ed4214be51519f4c != \1\b\d\9\4\c\4\d\b\1\9\3\7\a\3\2\e\d\4\2\1\4\b\e\5\1\5\1\9\f\4\c ]] 2024-09-22T00:59:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ cb879ab53bf9043914f34a512ea65640 != \c\b\8\7\9\a\b\5\3\b\f\9\0\4\3\9\1\4\f\3\4\a\5\1\2\e\a\6\5\6\4\0 ]] 2024-09-22T00:59:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:59:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 139025 2024-09-22T00:59:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:59:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 139027 2024-09-22T00:59:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=fcaa39fe33ca545582bd26df57acf268 2024-09-22T00:59:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-22T00:59:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ fcaa39fe33ca545582bd26df57acf268 != \f\c\a\a\3\9\f\e\3\3\c\a\5\4\5\5\8\2\b\d\2\6\d\f\5\7\a\c\f\2\6\8 ]] 2024-09-22T00:59:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=ff9c8732bbba5de009ea0ab831f19e34 2024-09-22T00:59:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-22T00:59:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ ff9c8732bbba5de009ea0ab831f19e34 != \f\f\9\c\8\7\3\2\b\b\b\a\5\d\e\0\0\9\e\a\0\a\b\8\3\1\f\1\9\e\3\4 ]] 2024-09-22T00:59:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=08cdf9bbd5d4ba3084809cefa26fb9b1 2024-09-22T00:59:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-22T00:59:44.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 08cdf9bbd5d4ba3084809cefa26fb9b1 != \0\8\c\d\f\9\b\b\d\5\d\4\b\a\3\0\8\4\8\0\9\c\e\f\a\2\6\f\b\9\b\1 ]] 2024-09-22T00:59:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ promote_md5=5ee574af460f345e9e6b13547cefde56 2024-09-22T00:59:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-22T00:59:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ [[ 5ee574af460f345e9e6b13547cefde56 != \5\e\e\5\7\4\a\f\4\6\0\f\3\4\5\e\9\e\6\b\1\3\5\4\7\c\e\f\d\e\5\6 ]] 2024-09-22T00:59:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:59:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 139029 2024-09-22T00:59:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:59:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 139030 2024-09-22T00:59:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:59:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 139032 2024-09-22T00:59:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:59:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 139034 2024-09-22T00:59:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:59:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 139037 2024-09-22T00:59:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:59:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 139038 2024-09-22T00:59:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-22T00:59:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait 139039 2024-09-22T00:59:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ (( RET != 0 )) 2024-09-22T00:59:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:59:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary1 2024-09-22T00:59:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-22T00:59:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:59:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:59:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-22T00:59:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:59:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:59:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-22T00:59:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:59:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:59:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-22T00:59:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary1: 2024-09-22T00:59:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 0f6d12b8-9468-4767-b8f3-f62a5392a241 2024-09-22T00:59:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:59:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3360153.6,"bytes_per_snapshot":67203072.0,"last_snapshot_bytes":67203072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966749,"remote_snapshot_timestamp":1726966749,"replay_state":"idle"} 2024-09-22T00:59:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:35 2024-09-22T00:59:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:59:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:59:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:59:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:40' 2024-09-22T00:59:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:59:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:59:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3360153.6,"bytes_per_snapshot":67203072.0,"last_snapshot_bytes":67203072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966749,"remote_snapshot_timestamp":1726966749,"replay_state":"idle"} 2024-09-22T00:59:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:59:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:59:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:35' 2024-09-22T00:59:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-22 00:59:35' 2024-09-22T00:59:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:59:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:59:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:35' 2024-09-22T00:59:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK1042614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary10f6d12b8-9468-4767-b8f3-f62a5392a241up+replayingreplaying, {"bytes_per_second":3360153.6,"bytes_per_snapshot":67203072.0,"last_snapshot_bytes":67203072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966749,"remote_snapshot_timestamp":1726966749,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45image-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+replayingreplaying, {"bytes_per_second":3256320.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+replayingreplaying, {"bytes_per_second":3325952.0,"bytes_per_snapshot":66519040.0,"last_snapshot_bytes":66519040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+replayingreplaying, {"bytes_per_second":3084902.4,"bytes_per_snapshot":61698048.0,"last_snapshot_bytes":61698048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966750,"remote_snapshot_timestamp":1726966750,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45image-primary4d8084146-8956-435b-a81f-43c80a014180up+replayingreplaying, {"bytes_per_second":3159040.0,"bytes_per_snapshot":63180800.0,"last_snapshot_bytes":63180800,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+replayingreplaying, {"bytes_per_second":3232768.0,"bytes_per_snapshot":64655360.0,"last_snapshot_bytes":64655360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+replayingreplaying, {"bytes_per_second":3317350.4,"bytes_per_snapshot":66347008.0,"last_snapshot_bytes":66347008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+replayingreplaying, {"bytes_per_second":3586252.8,"bytes_per_snapshot":71725056.0,"last_snapshot_bytes":71725056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966750,"remote_snapshot_timestamp":1726966750,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+replayingreplaying, {"bytes_per_second":3591168.0,"bytes_per_snapshot":71823360.0,"last_snapshot_bytes":71823360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+replayingreplaying, {"bytes_per_second":3247104.0,"bytes_per_snapshot":64942080.0,"last_snapshot_bytes":64942080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45' 2024-09-22T00:59:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-22T00:59:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:35' 2024-09-22T00:59:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-22T00:59:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:59:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:59:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:59:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:59:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-22T00:59:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:59:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-22T00:59:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-22T00:59:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:59:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-22T00:59:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary1 2024-09-22T00:59:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:59:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary1 2024-09-22T00:59:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary1 2024-09-22T00:59:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:59:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary2 2024-09-22T00:59:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-22T00:59:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:59:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:59:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-22T00:59:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:59:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:59:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-22T00:59:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:59:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:59:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-22T00:59:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary2: 2024-09-22T00:59:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: c7a13db2-2e62-40a8-86e3-e19844d44822 2024-09-22T00:59:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:59:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3325952.0,"bytes_per_snapshot":66519040.0,"last_snapshot_bytes":66519040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"} 2024-09-22T00:59:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:35 2024-09-22T00:59:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:59:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:59:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:59:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:45' 2024-09-22T00:59:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:59:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:59:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3325952.0,"bytes_per_snapshot":66519040.0,"last_snapshot_bytes":66519040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"} 2024-09-22T00:59:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:59:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:59:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:49.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:49.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:49.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:35' 2024-09-22T00:59:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-22 00:59:35' 2024-09-22T00:59:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:59:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:59:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:35' 2024-09-22T00:59:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK942614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+replayingreplaying, {"bytes_per_second":3256320.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45image-primary2c7a13db2-2e62-40a8-86e3-e19844d44822up+replayingreplaying, {"bytes_per_second":3325952.0,"bytes_per_snapshot":66519040.0,"last_snapshot_bytes":66519040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+replayingreplaying, {"bytes_per_second":3084902.4,"bytes_per_snapshot":61698048.0,"last_snapshot_bytes":61698048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966750,"remote_snapshot_timestamp":1726966750,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45image-primary4d8084146-8956-435b-a81f-43c80a014180up+replayingreplaying, {"bytes_per_second":3159040.0,"bytes_per_snapshot":63180800.0,"last_snapshot_bytes":63180800,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+replayingreplaying, {"bytes_per_second":3232768.0,"bytes_per_snapshot":64655360.0,"last_snapshot_bytes":64655360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+replayingreplaying, {"bytes_per_second":3317350.4,"bytes_per_snapshot":66347008.0,"last_snapshot_bytes":66347008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+replayingreplaying, {"bytes_per_second":3586252.8,"bytes_per_snapshot":71725056.0,"last_snapshot_bytes":71725056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966750,"remote_snapshot_timestamp":1726966750,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+replayingreplaying, {"bytes_per_second":3591168.0,"bytes_per_snapshot":71823360.0,"last_snapshot_bytes":71823360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+replayingreplaying, {"bytes_per_second":3247104.0,"bytes_per_snapshot":64942080.0,"last_snapshot_bytes":64942080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:45' 2024-09-22T00:59:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-22T00:59:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:35' 2024-09-22T00:59:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-22T00:59:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:59:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:59:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:59:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:59:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-22T00:59:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:59:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-22T00:59:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-22T00:59:50.527 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:59:50.524+0000 7f84953c5640 -1 reset not still connected to 0x5636f85f0d00 2024-09-22T00:59:50.535 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:59:50.528+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b32ea0 2024-09-22T00:59:50.535 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:59:50.528+0000 7f84953c5640 -1 reset not still connected to 0x5636f8b332b0 2024-09-22T00:59:50.535 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:59:50.528+0000 7f84953c5640 -1 reset not still connected to 0x5636f8ec80d0 2024-09-22T00:59:50.535 INFO:tasks.ceph.cluster2.osd.2.smithi060.stderr:2024-09-22T00:59:50.528+0000 7f84953c5640 -1 reset not still connected to 0x5636f9c289c0 2024-09-22T00:59:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:59:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-22T00:59:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary2 2024-09-22T00:59:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:59:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary2 2024-09-22T00:59:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary2 2024-09-22T00:59:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:59:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary3 2024-09-22T00:59:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-22T00:59:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:59:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:59:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-22T00:59:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:59:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:59:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-22T00:59:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:59:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:59:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-22T00:59:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary3: 2024-09-22T00:59:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 6175a8d9-0e7f-4d09-9b55-cc3a19b9272f 2024-09-22T00:59:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:59:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3084902.4,"bytes_per_snapshot":61698048.0,"last_snapshot_bytes":61698048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966750,"remote_snapshot_timestamp":1726966750,"replay_state":"idle"} 2024-09-22T00:59:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:35 2024-09-22T00:59:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:59:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:59:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:59:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:50' 2024-09-22T00:59:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:59:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:59:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3084902.4,"bytes_per_snapshot":61698048.0,"last_snapshot_bytes":61698048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966750,"remote_snapshot_timestamp":1726966750,"replay_state":"idle"} 2024-09-22T00:59:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:59:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:59:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:35' 2024-09-22T00:59:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-22 00:59:35' 2024-09-22T00:59:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:59:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:59:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:35' 2024-09-22T00:59:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK842614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+replayingreplaying, {"bytes_per_second":3256320.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:50image-primary36175a8d9-0e7f-4d09-9b55-cc3a19b9272fup+replayingreplaying, {"bytes_per_second":3084902.4,"bytes_per_snapshot":61698048.0,"last_snapshot_bytes":61698048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966750,"remote_snapshot_timestamp":1726966750,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:50image-primary4d8084146-8956-435b-a81f-43c80a014180up+replayingreplaying, {"bytes_per_second":3159040.0,"bytes_per_snapshot":63180800.0,"last_snapshot_bytes":63180800,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:50image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+replayingreplaying, {"bytes_per_second":3232768.0,"bytes_per_snapshot":64655360.0,"last_snapshot_bytes":64655360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:50image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+replayingreplaying, {"bytes_per_second":3317350.4,"bytes_per_snapshot":66347008.0,"last_snapshot_bytes":66347008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:50image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+replayingreplaying, {"bytes_per_second":3586252.8,"bytes_per_snapshot":71725056.0,"last_snapshot_bytes":71725056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966750,"remote_snapshot_timestamp":1726966750,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:50image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+replayingreplaying, {"bytes_per_second":3591168.0,"bytes_per_snapshot":71823360.0,"last_snapshot_bytes":71823360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:50image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+replayingreplaying, {"bytes_per_second":3247104.0,"bytes_per_snapshot":64942080.0,"last_snapshot_bytes":64942080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:50' 2024-09-22T00:59:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-22T00:59:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:35' 2024-09-22T00:59:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-22T00:59:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:59:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:59:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:59:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:59:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-22T00:59:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:59:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-22T00:59:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-22T00:59:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:59:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-22T00:59:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary3 2024-09-22T00:59:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:59:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary3 2024-09-22T00:59:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary3 2024-09-22T00:59:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:59:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary4 2024-09-22T00:59:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-22T00:59:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:59:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:59:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-22T00:59:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:59:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:59:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-22T00:59:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:59:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:59:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-22T00:59:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary4: 2024-09-22T00:59:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: d8084146-8956-435b-a81f-43c80a014180 2024-09-22T00:59:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:59:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3159040.0,"bytes_per_snapshot":63180800.0,"last_snapshot_bytes":63180800,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"} 2024-09-22T00:59:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:35 2024-09-22T00:59:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:59:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:59:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:59:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:50' 2024-09-22T00:59:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:59:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:59:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3159040.0,"bytes_per_snapshot":63180800.0,"last_snapshot_bytes":63180800,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"} 2024-09-22T00:59:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:59:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:59:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:35' 2024-09-22T00:59:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-22 00:59:35' 2024-09-22T00:59:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:59:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:59:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:35' 2024-09-22T00:59:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK742614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+replayingreplaying, {"bytes_per_second":3256320.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:50image-primary4d8084146-8956-435b-a81f-43c80a014180up+replayingreplaying, {"bytes_per_second":3159040.0,"bytes_per_snapshot":63180800.0,"last_snapshot_bytes":63180800,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:50image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+replayingreplaying, {"bytes_per_second":3232768.0,"bytes_per_snapshot":64655360.0,"last_snapshot_bytes":64655360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:50image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+replayingreplaying, {"bytes_per_second":3317350.4,"bytes_per_snapshot":66347008.0,"last_snapshot_bytes":66347008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:50image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+replayingreplaying, {"bytes_per_second":3586252.8,"bytes_per_snapshot":71725056.0,"last_snapshot_bytes":71725056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966750,"remote_snapshot_timestamp":1726966750,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:50image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+replayingreplaying, {"bytes_per_second":3591168.0,"bytes_per_snapshot":71823360.0,"last_snapshot_bytes":71823360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:50image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+replayingreplaying, {"bytes_per_second":3247104.0,"bytes_per_snapshot":64942080.0,"last_snapshot_bytes":64942080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:50' 2024-09-22T00:59:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-22T00:59:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:35' 2024-09-22T00:59:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-22T00:59:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:59:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:59:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:59:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:59:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-22T00:59:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:59:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-22T00:59:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-22T00:59:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T00:59:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-22T00:59:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary4 2024-09-22T00:59:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T00:59:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary4 2024-09-22T00:59:57.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:57.121 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary4 2024-09-22T00:59:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T00:59:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary5 2024-09-22T00:59:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-22T00:59:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:59:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T00:59:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-22T00:59:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:59:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:59:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T00:59:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T00:59:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-22T00:59:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:59:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:59:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T00:59:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T00:59:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-22T00:59:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary5: 2024-09-22T00:59:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 199f939a-c91f-4933-85bd-fed08290fd6e 2024-09-22T00:59:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T00:59:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3232768.0,"bytes_per_snapshot":64655360.0,"last_snapshot_bytes":64655360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"} 2024-09-22T00:59:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T00:59:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:35 2024-09-22T00:59:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T00:59:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T00:59:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T00:59:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T00:59:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:55' 2024-09-22T00:59:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T00:59:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T00:59:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T00:59:58.221 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3232768.0,"bytes_per_snapshot":64655360.0,"last_snapshot_bytes":64655360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"} 2024-09-22T00:59:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T00:59:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:59:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T00:59:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:59:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T00:59:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T00:59:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T00:59:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T00:59:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:35' 2024-09-22T00:59:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-22 00:59:35' 2024-09-22T00:59:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T00:59:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:59:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T00:59:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:35' 2024-09-22T00:59:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T00:59:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T00:59:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T00:59:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK642614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+replayingreplaying, {"bytes_per_second":3256320.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:55image-primary5199f939a-c91f-4933-85bd-fed08290fd6eup+replayingreplaying, {"bytes_per_second":3232768.0,"bytes_per_snapshot":64655360.0,"last_snapshot_bytes":64655360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:55image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+replayingreplaying, {"bytes_per_second":3317350.4,"bytes_per_snapshot":66347008.0,"last_snapshot_bytes":66347008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:55image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+replayingreplaying, {"bytes_per_second":3586252.8,"bytes_per_snapshot":71725056.0,"last_snapshot_bytes":71725056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966750,"remote_snapshot_timestamp":1726966750,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:55image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+replayingreplaying, {"bytes_per_second":3591168.0,"bytes_per_snapshot":71823360.0,"last_snapshot_bytes":71823360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:55image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+replayingreplaying, {"bytes_per_second":3247104.0,"bytes_per_snapshot":64942080.0,"last_snapshot_bytes":64942080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 00:59:55' 2024-09-22T00:59:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T00:59:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-22T00:59:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:35' 2024-09-22T00:59:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-22T00:59:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T00:59:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T00:59:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T00:59:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T00:59:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T00:59:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-22T00:59:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T00:59:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T00:59:58.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T00:59:58.386 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-22T00:59:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-22T01:00:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T01:00:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-22T01:00:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary5 2024-09-22T01:00:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T01:00:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T01:00:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary5 2024-09-22T01:00:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T01:00:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary5 2024-09-22T01:00:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T01:00:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary6 2024-09-22T01:00:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-22T01:00:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T01:00:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T01:00:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-22T01:00:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T01:00:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T01:00:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T01:00:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T01:00:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T01:00:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T01:00:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-22T01:00:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T01:00:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T01:00:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T01:00:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T01:00:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T01:00:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T01:00:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-22T01:00:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary6: 2024-09-22T01:00:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 4de603dc-70a4-4d8e-ac7a-d2f59bde04b9 2024-09-22T01:00:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T01:00:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3317350.4,"bytes_per_snapshot":66347008.0,"last_snapshot_bytes":66347008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"} 2024-09-22T01:00:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T01:00:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:35 2024-09-22T01:00:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T01:00:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T01:00:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T01:00:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T01:00:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 01:00:00' 2024-09-22T01:00:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T01:00:01.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T01:00:01.208 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T01:00:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3317350.4,"bytes_per_snapshot":66347008.0,"last_snapshot_bytes":66347008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"} 2024-09-22T01:00:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T01:00:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T01:00:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T01:00:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T01:00:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T01:00:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T01:00:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T01:00:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T01:00:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:35' 2024-09-22T01:00:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-22 00:59:35' 2024-09-22T01:00:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T01:00:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T01:00:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:35' 2024-09-22T01:00:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T01:00:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T01:00:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T01:00:01.325 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK542614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+replayingreplaying, {"bytes_per_second":3256320.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 01:00:00image-primary64de603dc-70a4-4d8e-ac7a-d2f59bde04b9up+replayingreplaying, {"bytes_per_second":3317350.4,"bytes_per_snapshot":66347008.0,"last_snapshot_bytes":66347008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 01:00:00image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+replayingreplaying, {"bytes_per_second":3586252.8,"bytes_per_snapshot":71725056.0,"last_snapshot_bytes":71725056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966750,"remote_snapshot_timestamp":1726966750,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 01:00:00image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+replayingreplaying, {"bytes_per_second":3591168.0,"bytes_per_snapshot":71823360.0,"last_snapshot_bytes":71823360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 01:00:00image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+replayingreplaying, {"bytes_per_second":3247104.0,"bytes_per_snapshot":64942080.0,"last_snapshot_bytes":64942080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 01:00:00' 2024-09-22T01:00:01.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T01:00:01.326 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-22T01:00:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:35' 2024-09-22T01:00:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-22T01:00:01.338 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T01:00:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T01:00:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T01:00:01.340 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T01:00:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T01:00:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T01:00:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-22T01:00:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T01:00:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T01:00:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-22T01:00:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-22T01:00:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T01:00:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-22T01:00:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary6 2024-09-22T01:00:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T01:00:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T01:00:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary6 2024-09-22T01:00:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T01:00:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary6 2024-09-22T01:00:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T01:00:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary7 2024-09-22T01:00:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-22T01:00:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T01:00:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T01:00:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-22T01:00:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T01:00:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T01:00:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T01:00:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T01:00:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T01:00:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T01:00:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-22T01:00:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T01:00:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T01:00:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T01:00:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T01:00:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T01:00:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T01:00:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-22T01:00:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary7: 2024-09-22T01:00:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: ec869a35-b7df-4a94-a692-4cf4ac04e207 2024-09-22T01:00:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T01:00:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":3586252.8,"bytes_per_snapshot":71725056.0,"last_snapshot_bytes":71725056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966750,"remote_snapshot_timestamp":1726966750,"replay_state":"idle"} 2024-09-22T01:00:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T01:00:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 00:59:35 2024-09-22T01:00:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T01:00:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T01:00:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T01:00:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T01:00:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 01:00:00' 2024-09-22T01:00:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T01:00:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T01:00:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T01:00:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":3586252.8,"bytes_per_snapshot":71725056.0,"last_snapshot_bytes":71725056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966750,"remote_snapshot_timestamp":1726966750,"replay_state":"idle"} 2024-09-22T01:00:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T01:00:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T01:00:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T01:00:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T01:00:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T01:00:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T01:00:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T01:00:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T01:00:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:35' 2024-09-22T01:00:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-22 00:59:35' 2024-09-22T01:00:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T01:00:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T01:00:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 00:59:35' 2024-09-22T01:00:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T01:00:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T01:00:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T01:00:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK442614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+replayingreplaying, {"bytes_per_second":3256320.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 01:00:00image-primary7ec869a35-b7df-4a94-a692-4cf4ac04e207up+replayingreplaying, {"bytes_per_second":3586252.8,"bytes_per_snapshot":71725056.0,"last_snapshot_bytes":71725056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966750,"remote_snapshot_timestamp":1726966750,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 01:00:00image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+replayingreplaying, {"bytes_per_second":3591168.0,"bytes_per_snapshot":71823360.0,"last_snapshot_bytes":71823360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 01:00:00image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+replayingreplaying, {"bytes_per_second":3247104.0,"bytes_per_snapshot":64942080.0,"last_snapshot_bytes":64942080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 00:59:35cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 01:00:00' 2024-09-22T01:00:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T01:00:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-22T01:00:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 00:59:35' 2024-09-22T01:00:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-22T01:00:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T01:00:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T01:00:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T01:00:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T01:00:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T01:00:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T01:00:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-22T01:00:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T01:00:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T01:00:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-22T01:00:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-22T01:00:06.131 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T01:00:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-22T01:00:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary7 2024-09-22T01:00:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T01:00:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T01:00:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary7 2024-09-22T01:00:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T01:00:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary7 2024-09-22T01:00:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T01:00:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary8 2024-09-22T01:00:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-22T01:00:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T01:00:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T01:00:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-22T01:00:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T01:00:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T01:00:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T01:00:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T01:00:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T01:00:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T01:00:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-22T01:00:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T01:00:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T01:00:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T01:00:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T01:00:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T01:00:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T01:00:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-22T01:00:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary8: 2024-09-22T01:00:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 078a49ba-3e14-42c8-acc1-269d8a4f54d9 2024-09-22T01:00:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T01:00:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71823360.0,"last_snapshot_bytes":71823360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"} 2024-09-22T01:00:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T01:00:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 01:00:05 2024-09-22T01:00:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T01:00:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T01:00:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T01:00:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T01:00:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 01:00:05' 2024-09-22T01:00:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T01:00:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T01:00:07.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T01:00:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71823360.0,"last_snapshot_bytes":71823360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"} 2024-09-22T01:00:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T01:00:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T01:00:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T01:00:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T01:00:07.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T01:00:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T01:00:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T01:00:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T01:00:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 01:00:05' 2024-09-22T01:00:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-22 01:00:05' 2024-09-22T01:00:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T01:00:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T01:00:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 01:00:05' 2024-09-22T01:00:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T01:00:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T01:00:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T01:00:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK342614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 01:00:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 01:00:05image-primary8078a49ba-3e14-42c8-acc1-269d8a4f54d9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71823360.0,"last_snapshot_bytes":71823360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 01:00:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 01:00:05image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64942080.0,"last_snapshot_bytes":64942080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 01:00:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 01:00:05' 2024-09-22T01:00:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T01:00:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-22T01:00:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 01:00:05' 2024-09-22T01:00:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-22T01:00:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T01:00:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T01:00:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T01:00:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T01:00:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T01:00:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T01:00:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-22T01:00:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T01:00:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T01:00:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-22T01:00:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-22T01:00:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T01:00:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-22T01:00:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary8 2024-09-22T01:00:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T01:00:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T01:00:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary8 2024-09-22T01:00:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T01:00:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary8 2024-09-22T01:00:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T01:00:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary9 2024-09-22T01:00:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-22T01:00:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T01:00:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T01:00:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-22T01:00:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T01:00:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T01:00:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T01:00:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T01:00:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T01:00:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T01:00:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-22T01:00:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T01:00:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T01:00:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T01:00:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T01:00:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T01:00:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T01:00:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-22T01:00:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary9: 2024-09-22T01:00:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 8102f56b-719c-4239-9d1a-177deba63ef3 2024-09-22T01:00:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T01:00:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64942080.0,"last_snapshot_bytes":64942080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"} 2024-09-22T01:00:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T01:00:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 01:00:05 2024-09-22T01:00:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T01:00:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T01:00:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T01:00:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T01:00:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 01:00:05' 2024-09-22T01:00:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T01:00:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T01:00:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T01:00:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64942080.0,"last_snapshot_bytes":64942080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"} 2024-09-22T01:00:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T01:00:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T01:00:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T01:00:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T01:00:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T01:00:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T01:00:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T01:00:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T01:00:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 01:00:05' 2024-09-22T01:00:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-22 01:00:05' 2024-09-22T01:00:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T01:00:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T01:00:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 01:00:05' 2024-09-22T01:00:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T01:00:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T01:00:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T01:00:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK242614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 01:00:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 01:00:05image-primary98102f56b-719c-4239-9d1a-177deba63ef3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64942080.0,"last_snapshot_bytes":64942080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 01:00:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 01:00:05' 2024-09-22T01:00:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T01:00:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-22T01:00:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 01:00:05' 2024-09-22T01:00:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-22T01:00:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T01:00:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T01:00:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T01:00:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T01:00:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T01:00:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T01:00:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-22T01:00:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T01:00:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T01:00:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-22T01:00:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-22T01:00:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T01:00:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-22T01:00:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary9 2024-09-22T01:00:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T01:00:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T01:00:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary9 2024-09-22T01:00:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T01:00:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary9 2024-09-22T01:00:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for j in {1..10} 2024-09-22T01:00:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ IMG=image-primary10 2024-09-22T01:00:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-22T01:00:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T01:00:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' snapshot = journal ']' 2024-09-22T01:00:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-22T01:00:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T01:00:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T01:00:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T01:00:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T01:00:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T01:00:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T01:00:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-22T01:00:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T01:00:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T01:00:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local description_pattern= 2024-09-22T01:00:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local service_pattern= 2024-09-22T01:00:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T01:00:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T01:00:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-22T01:00:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='image-primary10: 2024-09-22T01:00:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: global_id: 86d5828f-ec5b-4c64-81bd-fc524d8f94cf 2024-09-22T01:00:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+replaying 2024-09-22T01:00:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"} 2024-09-22T01:00:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: service: mirror.0 on smithi060 2024-09-22T01:00:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 01:00:05 2024-09-22T01:00:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: peer_sites: 2024-09-22T01:00:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: name: cluster2 2024-09-22T01:00:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: state: up+stopped 2024-09-22T01:00:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: description: local image is primary 2024-09-22T01:00:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: last_update: 2024-09-22 01:00:10' 2024-09-22T01:00:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ state: .*up+replaying' 2024-09-22T01:00:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: state: up+replaying 2024-09-22T01:00:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^ description: .*' 2024-09-22T01:00:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"} 2024-09-22T01:00:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n '' ']' 2024-09-22T01:00:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T01:00:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep '^up+' 2024-09-22T01:00:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T01:00:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep 'service: *mirror..* on ' 2024-09-22T01:00:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout: service: mirror.0 on smithi060 2024-09-22T01:00:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update 2024-09-22T01:00:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-22T01:00:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 01:00:05' 2024-09-22T01:00:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-22 01:00:05' 2024-09-22T01:00:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T01:00:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local state_pattern=up+replaying 2024-09-22T01:00:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local 'prev_last_update=2024-09-22 01:00:05' 2024-09-22T01:00:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local status 2024-09-22T01:00:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ CEPH_ARGS= 2024-09-22T01:00:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-22T01:00:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ status='OKOKOK142614264mirror.0smithi06019.1.1-373-g4e45e6b4trueOKimage-primary1086d5828f-ec5b-4c64-81bd-fc524d8f94cfup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726966748,"remote_snapshot_timestamp":1726966748,"replay_state":"idle"}42614264mirror.0smithi0602024-09-22 01:00:05cluster229740b4f-97f0-4f21-862e-dcc2872a1aadup+stoppedlocal image is primary2024-09-22 01:00:10' 2024-09-22T01:00:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local last_update state 2024-09-22T01:00:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-22T01:00:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ last_update='2024-09-22 01:00:05' 2024-09-22T01:00:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-22T01:00:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ state=up+replaying 2024-09-22T01:00:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo up+replaying 2024-09-22T01:00:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep up+replaying 2024-09-22T01:00:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:up+replaying 2024-09-22T01:00:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T01:00:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T01:00:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-22T01:00:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T01:00:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T01:00:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-22T01:00:13.448 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-22T01:00:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-22T01:00:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-22T01:00:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pool=mirror 2024-09-22T01:00:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local image=image-primary10 2024-09-22T01:00:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-22T01:00:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-22T01:00:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ grep -wq image-primary10 2024-09-22T01:00:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ return 0 2024-09-22T01:00:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -rf test-primary10 2024-09-22T01:00:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:OK 2024-09-22T01:00:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo OK 2024-09-22T01:00:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ cleanup 0 2024-09-22T01:00:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local error_code=0 2024-09-22T01:00:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ set +e 2024-09-22T01:00:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 0 -ne 0 ']' 2024-09-22T01:00:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -z '' ']' 2024-09-22T01:00:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster instance 2024-09-22T01:00:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-09-22T01:00:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-22T01:00:15.658 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T01:00:15.654+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281450 2024-09-22T01:00:15.658 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T01:00:15.654+0000 7f54f2d95640 -1 reset not still connected to 0x5617f3281520 2024-09-22T01:00:15.658 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T01:00:15.654+0000 7f54f2d95640 -1 reset not still connected to 0x5617f32815f0 2024-09-22T01:00:15.658 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2024-09-22T01:00:15.654+0000 7f54f2d95640 -1 reset not still connected to 0x5617f34ac0d0 2024-09-22T01:00:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:pool 'mirror' does not exist 2024-09-22T01:00:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-09-22T01:00:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-22T01:00:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:pool 'mirror' does not exist 2024-09-22T01:00:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-09-22T01:00:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-22T01:00:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:pool 'mirror_parent' does not exist 2024-09-22T01:00:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ CEPH_ARGS= 2024-09-22T01:00:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-22T01:00:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:pool 'mirror_parent' does not exist 2024-09-22T01:00:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-22T01:00:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stop_mirrors cluster1 2024-09-22T01:00:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1 2024-09-22T01:00:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local sig= 2024-09-22T01:00:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 0 0 2024-09-22T01:00:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-22T01:00:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stop_mirror cluster1:0 '' 2024-09-22T01:00:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster1:0 2024-09-22T01:00:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local sig= 2024-09-22T01:00:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n '' 2024-09-22T01:00:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pid 2024-09-22T01:00:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_pid_file cluster1:0 2024-09-22T01:00:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster1:0 2024-09-22T01:00:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-09-22T01:00:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2024-09-22T01:00:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster1:0 2024-09-22T01:00:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=cluster 2024-09-22T01:00:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-09-22T01:00:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-09-22T01:00:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-09-22T01:00:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-22T01:00:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval cluster=cluster1 2024-09-22T01:00:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ cluster=cluster1 2024-09-22T01:00:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-09-22T01:00:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-09-22T01:00:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-22T01:00:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-22T01:00:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-22T01:00:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ pid=30306 2024-09-22T01:00:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 30306 ']' 2024-09-22T01:00:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ kill 30306 2024-09-22T01:00:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 16 32 2024-09-22T01:00:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T01:00:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ps auxww 2024-09-22T01:00:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ awk -v pid=30306 '$2 == pid {print; exit 1}' 2024-09-22T01:00:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ break 2024-09-22T01:00:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ps auxww 2024-09-22T01:00:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ awk -v pid=30306 '$2 == pid {print; exit 1}' 2024-09-22T01:00:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-22T01:00:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local local_cluster=cluster1:0 2024-09-22T01:00:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-09-22T01:00:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-09-22T01:00:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-22T01:00:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-09-22T01:00:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=local_cluster 2024-09-22T01:00:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-09-22T01:00:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-09-22T01:00:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-09-22T01:00:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-22T01:00:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval local_cluster=cluster1 2024-09-22T01:00:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local_cluster=cluster1 2024-09-22T01:00:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-09-22T01:00:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-09-22T01:00:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-22T01:00:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-22T01:00:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-22T01:00:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ daemon_asok_file cluster1:0 cluster2 2024-09-22T01:00:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local local_cluster=cluster1:0 2024-09-22T01:00:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster2 2024-09-22T01:00:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-09-22T01:00:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-22T01:00:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-09-22T01:00:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=local_cluster 2024-09-22T01:00:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-09-22T01:00:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-09-22T01:00:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-09-22T01:00:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-22T01:00:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval local_cluster=cluster1 2024-09-22T01:00:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local_cluster=cluster1 2024-09-22T01:00:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-09-22T01:00:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-09-22T01:00:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-22T01:00:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-22T01:00:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-22T01:00:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ daemon_pid_file cluster1:0 2024-09-22T01:00:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1:0 2024-09-22T01:00:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-09-22T01:00:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-22T01:00:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster1:0 2024-09-22T01:00:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-09-22T01:00:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-09-22T01:00:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster1 2024-09-22T01:00:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-09-22T01:00:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-22T01:00:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster1 2024-09-22T01:00:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster1 2024-09-22T01:00:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-09-22T01:00:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-09-22T01:00:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-22T01:00:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-22T01:00:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-22T01:00:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-22T01:00:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stop_mirrors cluster2 2024-09-22T01:00:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2 2024-09-22T01:00:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local sig= 2024-09-22T01:00:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ seq 0 0 2024-09-22T01:00:20.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-22T01:00:20.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ stop_mirror cluster2:0 '' 2024-09-22T01:00:20.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local cluster=cluster2:0 2024-09-22T01:00:20.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local sig= 2024-09-22T01:00:20.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test -n '' 2024-09-22T01:00:20.172 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ local pid 2024-09-22T01:00:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ daemon_pid_file cluster2:0 2024-09-22T01:00:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster=cluster2:0 2024-09-22T01:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance 2024-09-22T01:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ set_cluster_instance cluster2:0 cluster instance 2024-09-22T01:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local val=cluster2:0 2024-09-22T01:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local cluster_var_name=cluster 2024-09-22T01:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local instance_var_name=instance 2024-09-22T01:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster2 2024-09-22T01:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-09-22T01:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-22T01:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval cluster=cluster2 2024-09-22T01:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ cluster=cluster2 2024-09-22T01:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ eval instance=0 2024-09-22T01:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ instance=0 2024-09-22T01:00:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-09-22T01:00:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-22T01:00:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-22T01:00:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ pid=30335 2024-09-22T01:00:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -n 30335 ']' 2024-09-22T01:00:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ kill 30335 2024-09-22T01:00:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ for s in 1 2 4 8 16 32 2024-09-22T01:00:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ sleep 1 2024-09-22T01:00:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ps auxww 2024-09-22T01:00:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ awk -v pid=30335 '$2 == pid {print; exit 1}' 2024-09-22T01:00:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ break 2024-09-22T01:00:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ ps auxww 2024-09-22T01:00:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ awk -v pid=30335 '$2 == pid {print; exit 1}' 2024-09-22T01:00:21.249 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ daemon_asok_file cluster2:0 cluster1 2024-09-22T01:00:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local local_cluster=cluster2:0 2024-09-22T01:00:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster1 2024-09-22T01:00:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-09-22T01:00:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-22T01:00:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster2:0 2024-09-22T01:00:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=local_cluster 2024-09-22T01:00:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-09-22T01:00:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster2 2024-09-22T01:00:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-09-22T01:00:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-22T01:00:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval local_cluster=cluster2 2024-09-22T01:00:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local_cluster=cluster2 2024-09-22T01:00:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-09-22T01:00:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-09-22T01:00:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-22T01:00:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-22T01:00:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-22T01:00:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-22T01:00:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local local_cluster=cluster2:0 2024-09-22T01:00:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster2 2024-09-22T01:00:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-09-22T01:00:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-22T01:00:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster2:0 2024-09-22T01:00:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=local_cluster 2024-09-22T01:00:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-09-22T01:00:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster2 2024-09-22T01:00:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-09-22T01:00:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-22T01:00:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval local_cluster=cluster2 2024-09-22T01:00:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ local_cluster=cluster2 2024-09-22T01:00:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-09-22T01:00:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-09-22T01:00:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-22T01:00:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-22T01:00:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-22T01:00:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ daemon_pid_file cluster2:0 2024-09-22T01:00:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster=cluster2:0 2024-09-22T01:00:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance 2024-09-22T01:00:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-22T01:00:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local val=cluster2:0 2024-09-22T01:00:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local cluster_var_name=cluster 2024-09-22T01:00:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ local instance_var_name=instance 2024-09-22T01:00:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ cluster=cluster2 2024-09-22T01:00:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ instance=0 2024-09-22T01:00:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-22T01:00:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval cluster=cluster2 2024-09-22T01:00:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ cluster=cluster2 2024-09-22T01:00:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ eval instance=0 2024-09-22T01:00:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ instance=0 2024-09-22T01:00:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-09-22T01:00:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-22T01:00:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-22T01:00:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' -z 1 ']' 2024-09-22T01:00:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ test '' = /tmp/tmp.5MnvHRcpWM 2024-09-22T01:00:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ rm -Rf /tmp/tmp.5MnvHRcpWM 2024-09-22T01:00:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ '[' 0 -eq 0 ']' 2024-09-22T01:00:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ echo OK 2024-09-22T01:00:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stdout:OK 2024-09-22T01:00:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi060.stderr:+ exit 0 2024-09-22T01:00:21.487 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-22T01:00:21.488 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-22T01:00:22.114 INFO:tasks.workunit:Stopping ['rbd/compare_mirror_images.sh'] on cluster1.client.mirror... 2024-09-22T01:00:22.114 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-22T01:00:22.987 DEBUG:teuthology.parallel:result is None 2024-09-22T01:00:22.987 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-22T01:00:23.012 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-22T01:00:23.012 DEBUG:teuthology.orchestra.run.smithi060:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-22T01:00:23.081 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-22T01:00:23.082 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-22T01:00:23.093 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-22T01:00:23.093 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-22T01:00:23.471 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T01:00:23.471 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-22T01:00:23.488 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":4017,"stamp":"2024-09-22T01:00:21.837636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":234,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2587212,"kb_used_data":18324,"kb_used_omap":4,"kb_used_meta":2568827,"kb_avail":280528308,"statfs":{"total":289910292480,"available":287260987392,"internally_reserved":0,"allocated":18763776,"data_stored":10926950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":2630479183},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-307651347,"num_objects":-120,"num_object_clones":-4,"num_object_copies":-240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-120,"num_whiteouts":0,"num_read":-713648,"num_read_kb":-36753666,"num_write":-150740,"num_write_kb":-29420865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11,"num_bytes_recovered":-4268722,"num_keys_recovered":-1,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.113720"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3778,"reported_epoch":1879,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.686981+0000","last_change":"2024-09-21T23:49:11.367983+0000","last_active":"2024-09-22T01:00:17.686981+0000","last_peered":"2024-09-22T01:00:17.686981+0000","last_clean":"2024-09-22T01:00:17.686981+0000","last_became_active":"2024-09-21T23:49:09.367547+0000","last_became_peered":"2024-09-21T23:49:09.367547+0000","last_unstale":"2024-09-22T01:00:17.686981+0000","last_undegraded":"2024-09-22T01:00:17.686981+0000","last_fullsized":"2024-09-22T01:00:17.686981+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:29:09.993126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00181346,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3778,"reported_epoch":1879,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.691077+0000","last_change":"2024-09-21T23:49:11.367837+0000","last_active":"2024-09-22T01:00:17.691077+0000","last_peered":"2024-09-22T01:00:17.691077+0000","last_clean":"2024-09-22T01:00:17.691077+0000","last_became_active":"2024-09-21T23:49:09.371121+0000","last_became_peered":"2024-09-21T23:49:09.371121+0000","last_unstale":"2024-09-22T01:00:17.691077+0000","last_undegraded":"2024-09-22T01:00:17.691077+0000","last_fullsized":"2024-09-22T01:00:17.691077+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:20:14.639018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018013790000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3780,"reported_epoch":1879,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.698745+0000","last_change":"2024-09-21T23:49:11.367216+0000","last_active":"2024-09-22T01:00:17.698745+0000","last_peered":"2024-09-22T01:00:17.698745+0000","last_clean":"2024-09-22T01:00:17.698745+0000","last_became_active":"2024-09-21T23:49:09.370827+0000","last_became_peered":"2024-09-21T23:49:09.370827+0000","last_unstale":"2024-09-22T01:00:17.698745+0000","last_undegraded":"2024-09-22T01:00:17.698745+0000","last_fullsized":"2024-09-22T01:00:17.698745+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:37:03.538645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011996000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3780,"reported_epoch":1879,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.686347+0000","last_change":"2024-09-21T23:49:11.368537+0000","last_active":"2024-09-22T01:00:17.686347+0000","last_peered":"2024-09-22T01:00:17.686347+0000","last_clean":"2024-09-22T01:00:17.686347+0000","last_became_active":"2024-09-21T23:49:09.363532+0000","last_became_peered":"2024-09-21T23:49:09.363532+0000","last_unstale":"2024-09-22T01:00:17.686347+0000","last_undegraded":"2024-09-22T01:00:17.686347+0000","last_fullsized":"2024-09-22T01:00:17.686347+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T11:41:34.822852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022342730000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":3785,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.759409+0000","last_change":"2024-09-21T23:49:11.376892+0000","last_active":"2024-09-22T01:00:18.759409+0000","last_peered":"2024-09-22T01:00:18.759409+0000","last_clean":"2024-09-22T01:00:18.759409+0000","last_became_active":"2024-09-21T23:49:09.365084+0000","last_became_peered":"2024-09-21T23:49:09.365084+0000","last_unstale":"2024-09-22T01:00:18.759409+0000","last_undegraded":"2024-09-22T01:00:18.759409+0000","last_fullsized":"2024-09-22T01:00:18.759409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:38:10.347807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021899129999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3783,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.760005+0000","last_change":"2024-09-21T23:49:11.657951+0000","last_active":"2024-09-22T01:00:18.760005+0000","last_peered":"2024-09-22T01:00:18.760005+0000","last_clean":"2024-09-22T01:00:18.760005+0000","last_became_active":"2024-09-21T23:49:09.368367+0000","last_became_peered":"2024-09-21T23:49:09.368367+0000","last_unstale":"2024-09-22T01:00:18.760005+0000","last_undegraded":"2024-09-22T01:00:18.760005+0000","last_fullsized":"2024-09-22T01:00:18.760005+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:23:41.547546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091250999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3784,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.758483+0000","last_change":"2024-09-21T23:49:11.657611+0000","last_active":"2024-09-22T01:00:18.758483+0000","last_peered":"2024-09-22T01:00:18.758483+0000","last_clean":"2024-09-22T01:00:18.758483+0000","last_became_active":"2024-09-21T23:49:09.366135+0000","last_became_peered":"2024-09-21T23:49:09.366135+0000","last_unstale":"2024-09-22T01:00:18.758483+0000","last_undegraded":"2024-09-22T01:00:18.758483+0000","last_fullsized":"2024-09-22T01:00:18.758483+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:06:48.237231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074559400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":3781,"reported_epoch":1879,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.690004+0000","last_change":"2024-09-21T23:49:11.367169+0000","last_active":"2024-09-22T01:00:17.690004+0000","last_peered":"2024-09-22T01:00:17.690004+0000","last_clean":"2024-09-22T01:00:17.690004+0000","last_became_active":"2024-09-21T23:49:09.365442+0000","last_became_peered":"2024-09-21T23:49:09.365442+0000","last_unstale":"2024-09-22T01:00:17.690004+0000","last_undegraded":"2024-09-22T01:00:17.690004+0000","last_fullsized":"2024-09-22T01:00:17.690004+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:30:16.716709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001193475,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"549'125","reported_seq":3943,"reported_epoch":1879,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.693267+0000","last_change":"2024-09-21T23:49:05.633330+0000","last_active":"2024-09-22T01:00:17.693267+0000","last_peered":"2024-09-22T01:00:17.693267+0000","last_clean":"2024-09-22T01:00:17.693267+0000","last_became_active":"2024-09-21T23:49:05.630853+0000","last_became_peered":"2024-09-21T23:49:05.630853+0000","last_unstale":"2024-09-22T01:00:17.693267+0000","last_undegraded":"2024-09-22T01:00:17.693267+0000","last_fullsized":"2024-09-22T01:00:17.693267+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:04.328167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:04.328167+0000","last_clean_scrub_stamp":"2024-09-21T23:49:04.328167+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:15:57.461205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739228,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":812964,"kb_used_data":5716,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":93558876,"statfs":{"total":96636764160,"available":95804289024,"internally_reserved":0,"allocated":5853184,"data_stored":3247491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739229,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":862128,"kb_used_data":6300,"kb_used_omap":1,"kb_used_meta":855806,"kb_avail":93509712,"statfs":{"total":96636764160,"available":95753945088,"internally_reserved":0,"allocated":6451200,"data_stored":3837878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":876345797},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739229,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":912120,"kb_used_data":6308,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":93459720,"statfs":{"total":96636764160,"available":95702753280,"internally_reserved":0,"allocated":6459392,"data_stored":3841581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":927529413},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T01:00:23.489 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-22T01:00:23.771 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T01:00:23.771 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-22T01:00:23.788 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":4017,"stamp":"2024-09-22T01:00:21.837636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":234,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2587212,"kb_used_data":18324,"kb_used_omap":4,"kb_used_meta":2568827,"kb_avail":280528308,"statfs":{"total":289910292480,"available":287260987392,"internally_reserved":0,"allocated":18763776,"data_stored":10926950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":2630479183},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-307651347,"num_objects":-120,"num_object_clones":-4,"num_object_copies":-240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-120,"num_whiteouts":0,"num_read":-713648,"num_read_kb":-36753666,"num_write":-150740,"num_write_kb":-29420865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11,"num_bytes_recovered":-4268722,"num_keys_recovered":-1,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.113720"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3778,"reported_epoch":1879,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.686981+0000","last_change":"2024-09-21T23:49:11.367983+0000","last_active":"2024-09-22T01:00:17.686981+0000","last_peered":"2024-09-22T01:00:17.686981+0000","last_clean":"2024-09-22T01:00:17.686981+0000","last_became_active":"2024-09-21T23:49:09.367547+0000","last_became_peered":"2024-09-21T23:49:09.367547+0000","last_unstale":"2024-09-22T01:00:17.686981+0000","last_undegraded":"2024-09-22T01:00:17.686981+0000","last_fullsized":"2024-09-22T01:00:17.686981+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:29:09.993126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00181346,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3778,"reported_epoch":1879,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.691077+0000","last_change":"2024-09-21T23:49:11.367837+0000","last_active":"2024-09-22T01:00:17.691077+0000","last_peered":"2024-09-22T01:00:17.691077+0000","last_clean":"2024-09-22T01:00:17.691077+0000","last_became_active":"2024-09-21T23:49:09.371121+0000","last_became_peered":"2024-09-21T23:49:09.371121+0000","last_unstale":"2024-09-22T01:00:17.691077+0000","last_undegraded":"2024-09-22T01:00:17.691077+0000","last_fullsized":"2024-09-22T01:00:17.691077+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:20:14.639018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018013790000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3780,"reported_epoch":1879,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.698745+0000","last_change":"2024-09-21T23:49:11.367216+0000","last_active":"2024-09-22T01:00:17.698745+0000","last_peered":"2024-09-22T01:00:17.698745+0000","last_clean":"2024-09-22T01:00:17.698745+0000","last_became_active":"2024-09-21T23:49:09.370827+0000","last_became_peered":"2024-09-21T23:49:09.370827+0000","last_unstale":"2024-09-22T01:00:17.698745+0000","last_undegraded":"2024-09-22T01:00:17.698745+0000","last_fullsized":"2024-09-22T01:00:17.698745+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:37:03.538645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011996000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3780,"reported_epoch":1879,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.686347+0000","last_change":"2024-09-21T23:49:11.368537+0000","last_active":"2024-09-22T01:00:17.686347+0000","last_peered":"2024-09-22T01:00:17.686347+0000","last_clean":"2024-09-22T01:00:17.686347+0000","last_became_active":"2024-09-21T23:49:09.363532+0000","last_became_peered":"2024-09-21T23:49:09.363532+0000","last_unstale":"2024-09-22T01:00:17.686347+0000","last_undegraded":"2024-09-22T01:00:17.686347+0000","last_fullsized":"2024-09-22T01:00:17.686347+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T11:41:34.822852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022342730000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":3785,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.759409+0000","last_change":"2024-09-21T23:49:11.376892+0000","last_active":"2024-09-22T01:00:18.759409+0000","last_peered":"2024-09-22T01:00:18.759409+0000","last_clean":"2024-09-22T01:00:18.759409+0000","last_became_active":"2024-09-21T23:49:09.365084+0000","last_became_peered":"2024-09-21T23:49:09.365084+0000","last_unstale":"2024-09-22T01:00:18.759409+0000","last_undegraded":"2024-09-22T01:00:18.759409+0000","last_fullsized":"2024-09-22T01:00:18.759409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:38:10.347807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021899129999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3783,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.760005+0000","last_change":"2024-09-21T23:49:11.657951+0000","last_active":"2024-09-22T01:00:18.760005+0000","last_peered":"2024-09-22T01:00:18.760005+0000","last_clean":"2024-09-22T01:00:18.760005+0000","last_became_active":"2024-09-21T23:49:09.368367+0000","last_became_peered":"2024-09-21T23:49:09.368367+0000","last_unstale":"2024-09-22T01:00:18.760005+0000","last_undegraded":"2024-09-22T01:00:18.760005+0000","last_fullsized":"2024-09-22T01:00:18.760005+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:23:41.547546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091250999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3784,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.758483+0000","last_change":"2024-09-21T23:49:11.657611+0000","last_active":"2024-09-22T01:00:18.758483+0000","last_peered":"2024-09-22T01:00:18.758483+0000","last_clean":"2024-09-22T01:00:18.758483+0000","last_became_active":"2024-09-21T23:49:09.366135+0000","last_became_peered":"2024-09-21T23:49:09.366135+0000","last_unstale":"2024-09-22T01:00:18.758483+0000","last_undegraded":"2024-09-22T01:00:18.758483+0000","last_fullsized":"2024-09-22T01:00:18.758483+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:06:48.237231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074559400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":3781,"reported_epoch":1879,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.690004+0000","last_change":"2024-09-21T23:49:11.367169+0000","last_active":"2024-09-22T01:00:17.690004+0000","last_peered":"2024-09-22T01:00:17.690004+0000","last_clean":"2024-09-22T01:00:17.690004+0000","last_became_active":"2024-09-21T23:49:09.365442+0000","last_became_peered":"2024-09-21T23:49:09.365442+0000","last_unstale":"2024-09-22T01:00:17.690004+0000","last_undegraded":"2024-09-22T01:00:17.690004+0000","last_fullsized":"2024-09-22T01:00:17.690004+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:30:16.716709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001193475,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"549'125","reported_seq":3943,"reported_epoch":1879,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.693267+0000","last_change":"2024-09-21T23:49:05.633330+0000","last_active":"2024-09-22T01:00:17.693267+0000","last_peered":"2024-09-22T01:00:17.693267+0000","last_clean":"2024-09-22T01:00:17.693267+0000","last_became_active":"2024-09-21T23:49:05.630853+0000","last_became_peered":"2024-09-21T23:49:05.630853+0000","last_unstale":"2024-09-22T01:00:17.693267+0000","last_undegraded":"2024-09-22T01:00:17.693267+0000","last_fullsized":"2024-09-22T01:00:17.693267+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:04.328167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:04.328167+0000","last_clean_scrub_stamp":"2024-09-21T23:49:04.328167+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:15:57.461205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739228,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":812964,"kb_used_data":5716,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":93558876,"statfs":{"total":96636764160,"available":95804289024,"internally_reserved":0,"allocated":5853184,"data_stored":3247491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739229,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":862128,"kb_used_data":6300,"kb_used_omap":1,"kb_used_meta":855806,"kb_avail":93509712,"statfs":{"total":96636764160,"available":95753945088,"internally_reserved":0,"allocated":6451200,"data_stored":3837878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":876345797},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739229,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":912120,"kb_used_data":6308,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":93459720,"statfs":{"total":96636764160,"available":95702753280,"internally_reserved":0,"allocated":6459392,"data_stored":3841581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":927529413},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T01:00:23.788 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-22T01:00:23.789 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-22T01:00:24.086 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T01:00:24.086 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-22T01:00:24.105 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":4018,"stamp":"2024-09-22T01:00:23.838104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2587224,"kb_used_data":18332,"kb_used_omap":4,"kb_used_meta":2568827,"kb_avail":280528296,"statfs":{"total":289910292480,"available":287260975104,"internally_reserved":0,"allocated":18771968,"data_stored":10930653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":2630479183},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-307651347,"num_objects":-120,"num_object_clones":-4,"num_object_copies":-240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-120,"num_whiteouts":0,"num_read":-713648,"num_read_kb":-36753666,"num_write":-150740,"num_write_kb":-29420865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11,"num_bytes_recovered":-4268722,"num_keys_recovered":-1,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001951"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3780,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.752338+0000","last_change":"2024-09-21T23:49:11.367983+0000","last_active":"2024-09-22T01:00:18.752338+0000","last_peered":"2024-09-22T01:00:18.752338+0000","last_clean":"2024-09-22T01:00:18.752338+0000","last_became_active":"2024-09-21T23:49:09.367547+0000","last_became_peered":"2024-09-21T23:49:09.367547+0000","last_unstale":"2024-09-22T01:00:18.752338+0000","last_undegraded":"2024-09-22T01:00:18.752338+0000","last_fullsized":"2024-09-22T01:00:18.752338+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:29:09.993126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00181346,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3780,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.762946+0000","last_change":"2024-09-21T23:49:11.367837+0000","last_active":"2024-09-22T01:00:18.762946+0000","last_peered":"2024-09-22T01:00:18.762946+0000","last_clean":"2024-09-22T01:00:18.762946+0000","last_became_active":"2024-09-21T23:49:09.371121+0000","last_became_peered":"2024-09-21T23:49:09.371121+0000","last_unstale":"2024-09-22T01:00:18.762946+0000","last_undegraded":"2024-09-22T01:00:18.762946+0000","last_fullsized":"2024-09-22T01:00:18.762946+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:20:14.639018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018013790000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3782,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.767162+0000","last_change":"2024-09-21T23:49:11.367216+0000","last_active":"2024-09-22T01:00:18.767162+0000","last_peered":"2024-09-22T01:00:18.767162+0000","last_clean":"2024-09-22T01:00:18.767162+0000","last_became_active":"2024-09-21T23:49:09.370827+0000","last_became_peered":"2024-09-21T23:49:09.370827+0000","last_unstale":"2024-09-22T01:00:18.767162+0000","last_undegraded":"2024-09-22T01:00:18.767162+0000","last_fullsized":"2024-09-22T01:00:18.767162+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:37:03.538645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011996000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3782,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.752709+0000","last_change":"2024-09-21T23:49:11.368537+0000","last_active":"2024-09-22T01:00:18.752709+0000","last_peered":"2024-09-22T01:00:18.752709+0000","last_clean":"2024-09-22T01:00:18.752709+0000","last_became_active":"2024-09-21T23:49:09.363532+0000","last_became_peered":"2024-09-21T23:49:09.363532+0000","last_unstale":"2024-09-22T01:00:18.752709+0000","last_undegraded":"2024-09-22T01:00:18.752709+0000","last_fullsized":"2024-09-22T01:00:18.752709+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T11:41:34.822852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022342730000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":3785,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.759409+0000","last_change":"2024-09-21T23:49:11.376892+0000","last_active":"2024-09-22T01:00:18.759409+0000","last_peered":"2024-09-22T01:00:18.759409+0000","last_clean":"2024-09-22T01:00:18.759409+0000","last_became_active":"2024-09-21T23:49:09.365084+0000","last_became_peered":"2024-09-21T23:49:09.365084+0000","last_unstale":"2024-09-22T01:00:18.759409+0000","last_undegraded":"2024-09-22T01:00:18.759409+0000","last_fullsized":"2024-09-22T01:00:18.759409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:38:10.347807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021899129999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3783,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.760005+0000","last_change":"2024-09-21T23:49:11.657951+0000","last_active":"2024-09-22T01:00:18.760005+0000","last_peered":"2024-09-22T01:00:18.760005+0000","last_clean":"2024-09-22T01:00:18.760005+0000","last_became_active":"2024-09-21T23:49:09.368367+0000","last_became_peered":"2024-09-21T23:49:09.368367+0000","last_unstale":"2024-09-22T01:00:18.760005+0000","last_undegraded":"2024-09-22T01:00:18.760005+0000","last_fullsized":"2024-09-22T01:00:18.760005+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:23:41.547546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091250999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3784,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.758483+0000","last_change":"2024-09-21T23:49:11.657611+0000","last_active":"2024-09-22T01:00:18.758483+0000","last_peered":"2024-09-22T01:00:18.758483+0000","last_clean":"2024-09-22T01:00:18.758483+0000","last_became_active":"2024-09-21T23:49:09.366135+0000","last_became_peered":"2024-09-21T23:49:09.366135+0000","last_unstale":"2024-09-22T01:00:18.758483+0000","last_undegraded":"2024-09-22T01:00:18.758483+0000","last_fullsized":"2024-09-22T01:00:18.758483+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:06:48.237231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074559400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":3783,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.760687+0000","last_change":"2024-09-21T23:49:11.367169+0000","last_active":"2024-09-22T01:00:18.760687+0000","last_peered":"2024-09-22T01:00:18.760687+0000","last_clean":"2024-09-22T01:00:18.760687+0000","last_became_active":"2024-09-21T23:49:09.365442+0000","last_became_peered":"2024-09-21T23:49:09.365442+0000","last_unstale":"2024-09-22T01:00:18.760687+0000","last_undegraded":"2024-09-22T01:00:18.760687+0000","last_fullsized":"2024-09-22T01:00:18.760687+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:30:16.716709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001193475,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"549'125","reported_seq":3945,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.750057+0000","last_change":"2024-09-21T23:49:05.633330+0000","last_active":"2024-09-22T01:00:18.750057+0000","last_peered":"2024-09-22T01:00:18.750057+0000","last_clean":"2024-09-22T01:00:18.750057+0000","last_became_active":"2024-09-21T23:49:05.630853+0000","last_became_peered":"2024-09-21T23:49:05.630853+0000","last_unstale":"2024-09-22T01:00:18.750057+0000","last_undegraded":"2024-09-22T01:00:18.750057+0000","last_fullsized":"2024-09-22T01:00:18.750057+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:04.328167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:04.328167+0000","last_clean_scrub_stamp":"2024-09-21T23:49:04.328167+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:15:57.461205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739228,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":812964,"kb_used_data":5716,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":93558876,"statfs":{"total":96636764160,"available":95804289024,"internally_reserved":0,"allocated":5853184,"data_stored":3247491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739230,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":862140,"kb_used_data":6308,"kb_used_omap":1,"kb_used_meta":855806,"kb_avail":93509700,"statfs":{"total":96636764160,"available":95753932800,"internally_reserved":0,"allocated":6459392,"data_stored":3841581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":876345797},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739229,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":912120,"kb_used_data":6308,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":93459720,"statfs":{"total":96636764160,"available":95702753280,"internally_reserved":0,"allocated":6459392,"data_stored":3841581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":927529413},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T01:00:24.105 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-22T01:00:24.399 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T01:00:24.400 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":1880,"fsid":"e1eca878-9fe9-437c-9e28-75a288e822bf","created":"2024-09-21T23:48:52.550274+0000","modified":"2024-09-22T01:00:18.735022+0000","last_up_change":"2024-09-21T23:49:02.706803+0000","last_in_change":"2024-09-21T23:48:53.468593+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T23:49:04.326544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T23:49:08.167148+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f4c56fb-237d-428d-9dcd-e02e7cb1904b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1391603771},{"type":"v1","addr":"172.21.15.60:6801","nonce":1391603771}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1391603771},{"type":"v1","addr":"172.21.15.60:6803","nonce":1391603771}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":1391603771},{"type":"v1","addr":"172.21.15.60:6807","nonce":1391603771}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":1391603771},{"type":"v1","addr":"172.21.15.60:6805","nonce":1391603771}]},"public_addr":"172.21.15.60:6801/1391603771","cluster_addr":"172.21.15.60:6803/1391603771","heartbeat_back_addr":"172.21.15.60:6807/1391603771","heartbeat_front_addr":"172.21.15.60:6805/1391603771","state":["exists","up"]},{"osd":1,"uuid":"6d00fac8-41ee-491e-b0d5-1ebe6d17078b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3160429338},{"type":"v1","addr":"172.21.15.60:6809","nonce":3160429338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3160429338},{"type":"v1","addr":"172.21.15.60:6811","nonce":3160429338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3160429338},{"type":"v1","addr":"172.21.15.60:6815","nonce":3160429338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3160429338},{"type":"v1","addr":"172.21.15.60:6813","nonce":3160429338}]},"public_addr":"172.21.15.60:6809/3160429338","cluster_addr":"172.21.15.60:6811/3160429338","heartbeat_back_addr":"172.21.15.60:6815/3160429338","heartbeat_front_addr":"172.21.15.60:6813/3160429338","state":["exists","up"]},{"osd":2,"uuid":"5ed7107f-a3cf-4376-8904-7c93e47acda5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":2394404319},{"type":"v1","addr":"172.21.15.60:6817","nonce":2394404319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":2394404319},{"type":"v1","addr":"172.21.15.60:6819","nonce":2394404319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":2394404319},{"type":"v1","addr":"172.21.15.60:6823","nonce":2394404319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2394404319},{"type":"v1","addr":"172.21.15.60:6821","nonce":2394404319}]},"public_addr":"172.21.15.60:6817/2394404319","cluster_addr":"172.21.15.60:6819/2394404319","heartbeat_back_addr":"172.21.15.60:6823/2394404319","heartbeat_front_addr":"172.21.15.60:6821/2394404319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:59.583845+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:59.538461+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:59.727579+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":3296,"length":1},{"begin":3301,"length":1},{"begin":3307,"length":1},{"begin":3310,"length":1},{"begin":3312,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T01:00:25.418 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-09-22T01:00:25.418 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-22T01:00:25.584 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-22T01:00:25.584 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-22T01:00:25.584 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-22T01:00:25.602 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-09-22T01:00:25.895 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 0 to deep-scrub 2024-09-22T01:00:25.913 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-09-22T01:00:25.914 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-22T01:00:26.066 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-22T01:00:26.066 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-22T01:00:26.067 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-22T01:00:26.083 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-09-22T01:00:26.388 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 1 to deep-scrub 2024-09-22T01:00:26.408 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-09-22T01:00:26.408 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-22T01:00:26.565 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-22T01:00:26.565 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-22T01:00:26.565 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-22T01:00:26.581 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-09-22T01:00:26.876 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 2 to deep-scrub 2024-09-22T01:00:26.895 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-22T01:00:27.185 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T01:00:27.185 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-22T01:00:27.202 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":4019,"stamp":"2024-09-22T01:00:25.838582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2587236,"kb_used_data":18340,"kb_used_omap":4,"kb_used_meta":2568827,"kb_avail":280528284,"statfs":{"total":289910292480,"available":287260962816,"internally_reserved":0,"allocated":18780160,"data_stored":10934356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":2630479183},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-5,"num_object_clones":0,"num_object_copies":-10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":0,"num_read":-2254,"num_read_kb":-2248,"num_write":-148,"num_write_kb":-146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-1,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.100804"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3780,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.752338+0000","last_change":"2024-09-21T23:49:11.367983+0000","last_active":"2024-09-22T01:00:18.752338+0000","last_peered":"2024-09-22T01:00:18.752338+0000","last_clean":"2024-09-22T01:00:18.752338+0000","last_became_active":"2024-09-21T23:49:09.367547+0000","last_became_peered":"2024-09-21T23:49:09.367547+0000","last_unstale":"2024-09-22T01:00:18.752338+0000","last_undegraded":"2024-09-22T01:00:18.752338+0000","last_fullsized":"2024-09-22T01:00:18.752338+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:29:09.993126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00181346,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3780,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.762946+0000","last_change":"2024-09-21T23:49:11.367837+0000","last_active":"2024-09-22T01:00:18.762946+0000","last_peered":"2024-09-22T01:00:18.762946+0000","last_clean":"2024-09-22T01:00:18.762946+0000","last_became_active":"2024-09-21T23:49:09.371121+0000","last_became_peered":"2024-09-21T23:49:09.371121+0000","last_unstale":"2024-09-22T01:00:18.762946+0000","last_undegraded":"2024-09-22T01:00:18.762946+0000","last_fullsized":"2024-09-22T01:00:18.762946+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:20:14.639018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018013790000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3782,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.767162+0000","last_change":"2024-09-21T23:49:11.367216+0000","last_active":"2024-09-22T01:00:18.767162+0000","last_peered":"2024-09-22T01:00:18.767162+0000","last_clean":"2024-09-22T01:00:18.767162+0000","last_became_active":"2024-09-21T23:49:09.370827+0000","last_became_peered":"2024-09-21T23:49:09.370827+0000","last_unstale":"2024-09-22T01:00:18.767162+0000","last_undegraded":"2024-09-22T01:00:18.767162+0000","last_fullsized":"2024-09-22T01:00:18.767162+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:37:03.538645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011996000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3782,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.752709+0000","last_change":"2024-09-21T23:49:11.368537+0000","last_active":"2024-09-22T01:00:18.752709+0000","last_peered":"2024-09-22T01:00:18.752709+0000","last_clean":"2024-09-22T01:00:18.752709+0000","last_became_active":"2024-09-21T23:49:09.363532+0000","last_became_peered":"2024-09-21T23:49:09.363532+0000","last_unstale":"2024-09-22T01:00:18.752709+0000","last_undegraded":"2024-09-22T01:00:18.752709+0000","last_fullsized":"2024-09-22T01:00:18.752709+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T11:41:34.822852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022342730000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":3785,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.759409+0000","last_change":"2024-09-21T23:49:11.376892+0000","last_active":"2024-09-22T01:00:18.759409+0000","last_peered":"2024-09-22T01:00:18.759409+0000","last_clean":"2024-09-22T01:00:18.759409+0000","last_became_active":"2024-09-21T23:49:09.365084+0000","last_became_peered":"2024-09-21T23:49:09.365084+0000","last_unstale":"2024-09-22T01:00:18.759409+0000","last_undegraded":"2024-09-22T01:00:18.759409+0000","last_fullsized":"2024-09-22T01:00:18.759409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:38:10.347807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021899129999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3783,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.760005+0000","last_change":"2024-09-21T23:49:11.657951+0000","last_active":"2024-09-22T01:00:18.760005+0000","last_peered":"2024-09-22T01:00:18.760005+0000","last_clean":"2024-09-22T01:00:18.760005+0000","last_became_active":"2024-09-21T23:49:09.368367+0000","last_became_peered":"2024-09-21T23:49:09.368367+0000","last_unstale":"2024-09-22T01:00:18.760005+0000","last_undegraded":"2024-09-22T01:00:18.760005+0000","last_fullsized":"2024-09-22T01:00:18.760005+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:23:41.547546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091250999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3784,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.758483+0000","last_change":"2024-09-21T23:49:11.657611+0000","last_active":"2024-09-22T01:00:18.758483+0000","last_peered":"2024-09-22T01:00:18.758483+0000","last_clean":"2024-09-22T01:00:18.758483+0000","last_became_active":"2024-09-21T23:49:09.366135+0000","last_became_peered":"2024-09-21T23:49:09.366135+0000","last_unstale":"2024-09-22T01:00:18.758483+0000","last_undegraded":"2024-09-22T01:00:18.758483+0000","last_fullsized":"2024-09-22T01:00:18.758483+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:06:48.237231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074559400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":3783,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.760687+0000","last_change":"2024-09-21T23:49:11.367169+0000","last_active":"2024-09-22T01:00:18.760687+0000","last_peered":"2024-09-22T01:00:18.760687+0000","last_clean":"2024-09-22T01:00:18.760687+0000","last_became_active":"2024-09-21T23:49:09.365442+0000","last_became_peered":"2024-09-21T23:49:09.365442+0000","last_unstale":"2024-09-22T01:00:18.760687+0000","last_undegraded":"2024-09-22T01:00:18.760687+0000","last_fullsized":"2024-09-22T01:00:18.760687+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:08.344894+0000","last_clean_scrub_stamp":"2024-09-21T23:49:08.344894+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:30:16.716709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001193475,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"549'125","reported_seq":3945,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:18.750057+0000","last_change":"2024-09-21T23:49:05.633330+0000","last_active":"2024-09-22T01:00:18.750057+0000","last_peered":"2024-09-22T01:00:18.750057+0000","last_clean":"2024-09-22T01:00:18.750057+0000","last_became_active":"2024-09-21T23:49:05.630853+0000","last_became_peered":"2024-09-21T23:49:05.630853+0000","last_unstale":"2024-09-22T01:00:18.750057+0000","last_undegraded":"2024-09-22T01:00:18.750057+0000","last_fullsized":"2024-09-22T01:00:18.750057+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:49:04.328167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:49:04.328167+0000","last_clean_scrub_stamp":"2024-09-21T23:49:04.328167+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:15:57.461205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739229,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":812976,"kb_used_data":5724,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":93558864,"statfs":{"total":96636764160,"available":95804276736,"internally_reserved":0,"allocated":5861376,"data_stored":3251194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739230,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":862140,"kb_used_data":6308,"kb_used_omap":1,"kb_used_meta":855806,"kb_avail":93509700,"statfs":{"total":96636764160,"available":95753932800,"internally_reserved":0,"allocated":6459392,"data_stored":3841581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":876345797},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739230,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":912120,"kb_used_data":6308,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":93459720,"statfs":{"total":96636764160,"available":95702753280,"internally_reserved":0,"allocated":6459392,"data_stored":3841581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":927529413},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T01:00:27.205 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-21T23:49:08.344894+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=49, tm_sec=8, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=1, tm_min=0, tm_sec=24, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T01:00:27.205 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-21T23:49:08.344894+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=49, tm_sec=8, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=1, tm_min=0, tm_sec=24, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T01:00:27.205 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-21T23:49:08.344894+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=49, tm_sec=8, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=1, tm_min=0, tm_sec=24, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T01:00:27.205 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-21T23:49:08.344894+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=49, tm_sec=8, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=1, tm_min=0, tm_sec=24, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T01:00:27.205 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-21T23:49:08.344894+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=49, tm_sec=8, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=1, tm_min=0, tm_sec=24, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T01:00:27.205 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-21T23:49:08.344894+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=49, tm_sec=8, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=1, tm_min=0, tm_sec=24, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T01:00:27.205 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-21T23:49:08.344894+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=49, tm_sec=8, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=1, tm_min=0, tm_sec=24, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T01:00:27.206 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-21T23:49:08.344894+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=49, tm_sec=8, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=1, tm_min=0, tm_sec=24, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T01:00:27.206 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-21T23:49:04.328167+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=49, tm_sec=4, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=1, tm_min=0, tm_sec=24, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T01:00:27.206 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-22T01:00:47.207 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-22T01:00:47.519 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T01:00:47.519 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-22T01:00:47.539 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":4029,"stamp":"2024-09-22T01:00:45.843460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2587220,"kb_used_data":18340,"kb_used_omap":4,"kb_used_meta":2568827,"kb_avail":280528300,"statfs":{"total":289910292480,"available":287260979200,"internally_reserved":0,"allocated":18780160,"data_stored":10934356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":2630479183},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002938"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3790,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:30.073883+0000","last_change":"2024-09-22T01:00:30.073883+0000","last_active":"2024-09-22T01:00:30.073883+0000","last_peered":"2024-09-22T01:00:30.073883+0000","last_clean":"2024-09-22T01:00:30.073883+0000","last_became_active":"2024-09-21T23:49:09.367547+0000","last_became_peered":"2024-09-21T23:49:09.367547+0000","last_unstale":"2024-09-22T01:00:30.073883+0000","last_undegraded":"2024-09-22T01:00:30.073883+0000","last_fullsized":"2024-09-22T01:00:30.073883+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T01:00:30.073624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T01:00:30.073624+0000","last_clean_scrub_stamp":"2024-09-22T01:00:30.073624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:20:29.801929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00181346,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3790,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:32.103279+0000","last_change":"2024-09-22T01:00:32.103279+0000","last_active":"2024-09-22T01:00:32.103279+0000","last_peered":"2024-09-22T01:00:32.103279+0000","last_clean":"2024-09-22T01:00:32.103279+0000","last_became_active":"2024-09-21T23:49:09.371121+0000","last_became_peered":"2024-09-21T23:49:09.371121+0000","last_unstale":"2024-09-22T01:00:32.103279+0000","last_undegraded":"2024-09-22T01:00:32.103279+0000","last_fullsized":"2024-09-22T01:00:32.103279+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T01:00:32.103018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T01:00:32.103018+0000","last_clean_scrub_stamp":"2024-09-22T01:00:32.103018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:33:59.040491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0018013790000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3791,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:31.101337+0000","last_change":"2024-09-22T01:00:31.101395+0000","last_active":"2024-09-22T01:00:31.101337+0000","last_peered":"2024-09-22T01:00:31.101337+0000","last_clean":"2024-09-22T01:00:31.101337+0000","last_became_active":"2024-09-21T23:49:09.370827+0000","last_became_peered":"2024-09-21T23:49:09.370827+0000","last_unstale":"2024-09-22T01:00:31.101337+0000","last_undegraded":"2024-09-22T01:00:31.101337+0000","last_fullsized":"2024-09-22T01:00:31.101337+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T01:00:31.101108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T01:00:31.101108+0000","last_clean_scrub_stamp":"2024-09-22T01:00:31.101108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T11:14:41.347697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011996000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3791,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:28.089515+0000","last_change":"2024-09-22T01:00:28.089691+0000","last_active":"2024-09-22T01:00:28.089515+0000","last_peered":"2024-09-22T01:00:28.089515+0000","last_clean":"2024-09-22T01:00:28.089515+0000","last_became_active":"2024-09-21T23:49:09.363532+0000","last_became_peered":"2024-09-21T23:49:09.363532+0000","last_unstale":"2024-09-22T01:00:28.089515+0000","last_undegraded":"2024-09-22T01:00:28.089515+0000","last_fullsized":"2024-09-22T01:00:28.089515+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T01:00:28.089278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T01:00:28.089278+0000","last_clean_scrub_stamp":"2024-09-22T01:00:28.089278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:53:59.953056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022342730000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"1880'3","reported_seq":3795,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:25.933646+0000","last_change":"2024-09-22T01:00:25.933668+0000","last_active":"2024-09-22T01:00:25.933646+0000","last_peered":"2024-09-22T01:00:25.933646+0000","last_clean":"2024-09-22T01:00:25.933646+0000","last_became_active":"2024-09-21T23:49:09.365084+0000","last_became_peered":"2024-09-21T23:49:09.365084+0000","last_unstale":"2024-09-22T01:00:25.933646+0000","last_undegraded":"2024-09-22T01:00:25.933646+0000","last_fullsized":"2024-09-22T01:00:25.933646+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"1880'3","last_scrub_stamp":"2024-09-22T01:00:25.933579+0000","last_deep_scrub":"1880'3","last_deep_scrub_stamp":"2024-09-22T01:00:25.933579+0000","last_clean_scrub_stamp":"2024-09-22T01:00:25.933579+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:05:34.824960+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0021899129999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3792,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:28.609206+0000","last_change":"2024-09-22T01:00:28.609321+0000","last_active":"2024-09-22T01:00:28.609206+0000","last_peered":"2024-09-22T01:00:28.609206+0000","last_clean":"2024-09-22T01:00:28.609206+0000","last_became_active":"2024-09-21T23:49:09.368367+0000","last_became_peered":"2024-09-21T23:49:09.368367+0000","last_unstale":"2024-09-22T01:00:28.609206+0000","last_undegraded":"2024-09-22T01:00:28.609206+0000","last_fullsized":"2024-09-22T01:00:28.609206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T01:00:28.608942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T01:00:28.608942+0000","last_clean_scrub_stamp":"2024-09-22T01:00:28.608942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T04:08:23.715363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091250999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3793,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:27.654194+0000","last_change":"2024-09-22T01:00:27.654245+0000","last_active":"2024-09-22T01:00:27.654194+0000","last_peered":"2024-09-22T01:00:27.654194+0000","last_clean":"2024-09-22T01:00:27.654194+0000","last_became_active":"2024-09-21T23:49:09.366135+0000","last_became_peered":"2024-09-21T23:49:09.366135+0000","last_unstale":"2024-09-22T01:00:27.654194+0000","last_undegraded":"2024-09-22T01:00:27.654194+0000","last_fullsized":"2024-09-22T01:00:27.654194+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T01:00:27.653991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T01:00:27.653991+0000","last_clean_scrub_stamp":"2024-09-22T01:00:27.653991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:30:03.404717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00074559400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"1880'2","reported_seq":3793,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:29.046527+0000","last_change":"2024-09-22T01:00:29.046704+0000","last_active":"2024-09-22T01:00:29.046527+0000","last_peered":"2024-09-22T01:00:29.046527+0000","last_clean":"2024-09-22T01:00:29.046527+0000","last_became_active":"2024-09-21T23:49:09.365442+0000","last_became_peered":"2024-09-21T23:49:09.365442+0000","last_unstale":"2024-09-22T01:00:29.046527+0000","last_undegraded":"2024-09-22T01:00:29.046527+0000","last_fullsized":"2024-09-22T01:00:29.046527+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"1880'2","last_scrub_stamp":"2024-09-22T01:00:29.046295+0000","last_deep_scrub":"1880'2","last_deep_scrub_stamp":"2024-09-22T01:00:29.046295+0000","last_clean_scrub_stamp":"2024-09-22T01:00:29.046295+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:16:59.302644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001193475,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"1880'127","reported_seq":3956,"reported_epoch":1880,"state":"active+clean","last_fresh":"2024-09-22T01:00:27.066953+0000","last_change":"2024-09-22T01:00:27.067033+0000","last_active":"2024-09-22T01:00:27.066953+0000","last_peered":"2024-09-22T01:00:27.066953+0000","last_clean":"2024-09-22T01:00:27.066953+0000","last_became_active":"2024-09-21T23:49:05.630853+0000","last_became_peered":"2024-09-21T23:49:05.630853+0000","last_unstale":"2024-09-22T01:00:27.066953+0000","last_undegraded":"2024-09-22T01:00:27.066953+0000","last_fullsized":"2024-09-22T01:00:27.066953+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"1880'127","last_scrub_stamp":"2024-09-22T01:00:27.066813+0000","last_deep_scrub":"1880'127","last_deep_scrub_stamp":"2024-09-22T01:00:27.066813+0000","last_clean_scrub_stamp":"2024-09-22T01:00:27.066813+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:52:03.196095+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739233,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":812972,"kb_used_data":5724,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":93558868,"statfs":{"total":96636764160,"available":95804280832,"internally_reserved":0,"allocated":5861376,"data_stored":3251194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739234,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":862132,"kb_used_data":6308,"kb_used_omap":1,"kb_used_meta":855806,"kb_avail":93509708,"statfs":{"total":96636764160,"available":95753940992,"internally_reserved":0,"allocated":6459392,"data_stored":3841581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":876345797},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739234,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":912116,"kb_used_data":6308,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":93459724,"statfs":{"total":96636764160,"available":95702757376,"internally_reserved":0,"allocated":6459392,"data_stored":3841581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":927529413},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T01:00:47.541 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-09-22T01:00:47.864 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-22T01:00:47.864 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-22T01:00:47.864 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-22T01:00:47.864 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T01:00:50.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2024-09-22T01:00:53.966 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-22T01:00:53.966 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-22T01:00:53.966 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T01:00:56.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~0s 2024-09-22T01:00:56.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 has been restored 2024-09-22T01:01:00.068 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-22T01:01:00.068 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-22T01:01:00.068 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T01:01:01.427 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 is failed for ~0s 2024-09-22T01:01:01.427 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 has been restored 2024-09-22T01:01:06.170 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-22T01:01:06.170 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-22T01:01:06.170 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-22T01:01:06.170 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T01:01:06.217 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-22T01:01:06.217 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-22T01:01:06.217 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-22T01:01:06.217 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T01:01:06.251 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-22T01:01:06.251 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-22T01:01:06.252 DEBUG:teuthology.orchestra.run.smithi060:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-22T01:01:06.373 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-22T01:01:06.373 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-22T01:01:06.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 has been restored 2024-09-22T01:01:06.572 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-22T01:01:06.572 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-22T01:01:06.769 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-22T01:01:06.774 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-22T01:01:06.977 INFO:tasks.ceph:Archiving mon data... 2024-09-22T01:01:06.979 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914544/data/cluster2.mon.a.tgz 2024-09-22T01:01:06.979 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-09-22T01:01:06.984 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.EwGje9fmUa 2024-09-22T01:01:06.985 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.EwGje9fmUa 2024-09-22T01:01:07.181 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.EwGje9fmUa 2024-09-22T01:01:07.253 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.EwGje9fmUa is 553KB 2024-09-22T01:01:07.315 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.EwGje9fmUa 2024-09-22T01:01:07.320 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-22T01:01:07.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-22T01:01:07.323 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-22T01:01:07.540 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-22T01:01:07.549 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-22T01:01:07.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-22T01:01:08.112 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-22T01:01:08.112 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-22T01:01:08.153 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":4075,"stamp":"2024-09-22T01:01:08.112226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3819976,"kb_used_data":14092,"kb_used_omap":4,"kb_used_meta":3805819,"kb_avail":279295544,"statfs":{"total":289910292480,"available":285998637056,"internally_reserved":0,"allocated":14430208,"data_stored":8453968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":3897158992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002939"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3814,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.878316+0000","last_change":"2024-09-21T23:48:32.523721+0000","last_active":"2024-09-22T01:00:17.878316+0000","last_peered":"2024-09-22T01:00:17.878316+0000","last_clean":"2024-09-22T01:00:17.878316+0000","last_became_active":"2024-09-21T23:48:30.498942+0000","last_became_peered":"2024-09-21T23:48:30.498942+0000","last_unstale":"2024-09-22T01:00:17.878316+0000","last_undegraded":"2024-09-22T01:00:17.878316+0000","last_fullsized":"2024-09-22T01:00:17.878316+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:55:17.700760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019891520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3814,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.880127+0000","last_change":"2024-09-21T23:48:32.522974+0000","last_active":"2024-09-22T01:00:17.880127+0000","last_peered":"2024-09-22T01:00:17.880127+0000","last_clean":"2024-09-22T01:00:17.880127+0000","last_became_active":"2024-09-21T23:48:30.499944+0000","last_became_peered":"2024-09-21T23:48:30.499944+0000","last_unstale":"2024-09-22T01:00:17.880127+0000","last_undegraded":"2024-09-22T01:00:17.880127+0000","last_fullsized":"2024-09-22T01:00:17.880127+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:40:33.520270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016043909999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3815,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.878552+0000","last_change":"2024-09-21T23:48:32.522522+0000","last_active":"2024-09-22T01:00:17.878552+0000","last_peered":"2024-09-22T01:00:17.878552+0000","last_clean":"2024-09-22T01:00:17.878552+0000","last_became_active":"2024-09-21T23:48:30.498992+0000","last_became_peered":"2024-09-21T23:48:30.498992+0000","last_unstale":"2024-09-22T01:00:17.878552+0000","last_undegraded":"2024-09-22T01:00:17.878552+0000","last_fullsized":"2024-09-22T01:00:17.878552+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:30:17.385732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001283916,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3815,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.877569+0000","last_change":"2024-09-21T23:48:32.522399+0000","last_active":"2024-09-22T01:00:17.877569+0000","last_peered":"2024-09-22T01:00:17.877569+0000","last_clean":"2024-09-22T01:00:17.877569+0000","last_became_active":"2024-09-21T23:48:30.499210+0000","last_became_peered":"2024-09-21T23:48:30.499210+0000","last_unstale":"2024-09-22T01:00:17.877569+0000","last_undegraded":"2024-09-22T01:00:17.877569+0000","last_fullsized":"2024-09-22T01:00:17.877569+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:38:47.969858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012027369999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":3817,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:19.474795+0000","last_change":"2024-09-21T23:48:32.518542+0000","last_active":"2024-09-22T01:00:19.474795+0000","last_peered":"2024-09-22T01:00:19.474795+0000","last_clean":"2024-09-22T01:00:19.474795+0000","last_became_active":"2024-09-21T23:48:30.498643+0000","last_became_peered":"2024-09-21T23:48:30.498643+0000","last_unstale":"2024-09-22T01:00:19.474795+0000","last_undegraded":"2024-09-22T01:00:19.474795+0000","last_fullsized":"2024-09-22T01:00:19.474795+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:45:35.695733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019200619999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3815,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:19.363604+0000","last_change":"2024-09-21T23:48:32.513550+0000","last_active":"2024-09-22T01:00:19.363604+0000","last_peered":"2024-09-22T01:00:19.363604+0000","last_clean":"2024-09-22T01:00:19.363604+0000","last_became_active":"2024-09-21T23:48:30.500705+0000","last_became_peered":"2024-09-21T23:48:30.500705+0000","last_unstale":"2024-09-22T01:00:19.363604+0000","last_undegraded":"2024-09-22T01:00:19.363604+0000","last_fullsized":"2024-09-22T01:00:19.363604+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:21:01.132226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096276900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3817,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:19.364098+0000","last_change":"2024-09-21T23:48:32.513385+0000","last_active":"2024-09-22T01:00:19.364098+0000","last_peered":"2024-09-22T01:00:19.364098+0000","last_clean":"2024-09-22T01:00:19.364098+0000","last_became_active":"2024-09-21T23:48:30.501645+0000","last_became_peered":"2024-09-21T23:48:30.501645+0000","last_unstale":"2024-09-22T01:00:19.364098+0000","last_undegraded":"2024-09-22T01:00:19.364098+0000","last_fullsized":"2024-09-22T01:00:19.364098+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:12:39.920716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093488900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":3816,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.877374+0000","last_change":"2024-09-21T23:48:32.523316+0000","last_active":"2024-09-22T01:00:17.877374+0000","last_peered":"2024-09-22T01:00:17.877374+0000","last_clean":"2024-09-22T01:00:17.877374+0000","last_became_active":"2024-09-21T23:48:30.501010+0000","last_became_peered":"2024-09-21T23:48:30.501010+0000","last_unstale":"2024-09-22T01:00:17.877374+0000","last_undegraded":"2024-09-22T01:00:17.877374+0000","last_fullsized":"2024-09-22T01:00:17.877374+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:40:07.811058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001659017,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"556'125","reported_seq":3979,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.876769+0000","last_change":"2024-09-21T23:48:26.479535+0000","last_active":"2024-09-22T01:00:17.876769+0000","last_peered":"2024-09-22T01:00:17.876769+0000","last_clean":"2024-09-22T01:00:17.876769+0000","last_became_active":"2024-09-21T23:48:26.475455+0000","last_became_peered":"2024-09-21T23:48:26.475455+0000","last_unstale":"2024-09-22T01:00:17.876769+0000","last_undegraded":"2024-09-22T01:00:17.876769+0000","last_fullsized":"2024-09-22T01:00:17.876769+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:25.462486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:25.462486+0000","last_clean_scrub_stamp":"2024-09-21T23:48:25.462486+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T04:57:09.281011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739245,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1266920,"kb_used_data":4308,"kb_used_omap":1,"kb_used_meta":1262590,"kb_avail":93104920,"statfs":{"total":96636764160,"available":95339438080,"internally_reserved":0,"allocated":4411392,"data_stored":2424398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1292892613},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739245,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1191152,"kb_used_data":4892,"kb_used_omap":1,"kb_used_meta":1186238,"kb_avail":93180688,"statfs":{"total":96636764160,"available":95417024512,"internally_reserved":0,"allocated":5009408,"data_stored":3014785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1214708165},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739247,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1361904,"kb_used_data":4892,"kb_used_omap":1,"kb_used_meta":1356990,"kb_avail":93009936,"statfs":{"total":96636764160,"available":95242174464,"internally_reserved":0,"allocated":5009408,"data_stored":3014785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1389558214},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T01:01:08.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-22T01:01:08.466 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-22T01:01:08.467 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-22T01:01:08.486 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":4075,"stamp":"2024-09-22T01:01:08.112226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3819976,"kb_used_data":14092,"kb_used_omap":4,"kb_used_meta":3805819,"kb_avail":279295544,"statfs":{"total":289910292480,"available":285998637056,"internally_reserved":0,"allocated":14430208,"data_stored":8453968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":3897158992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002939"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3814,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.878316+0000","last_change":"2024-09-21T23:48:32.523721+0000","last_active":"2024-09-22T01:00:17.878316+0000","last_peered":"2024-09-22T01:00:17.878316+0000","last_clean":"2024-09-22T01:00:17.878316+0000","last_became_active":"2024-09-21T23:48:30.498942+0000","last_became_peered":"2024-09-21T23:48:30.498942+0000","last_unstale":"2024-09-22T01:00:17.878316+0000","last_undegraded":"2024-09-22T01:00:17.878316+0000","last_fullsized":"2024-09-22T01:00:17.878316+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:55:17.700760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019891520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3814,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.880127+0000","last_change":"2024-09-21T23:48:32.522974+0000","last_active":"2024-09-22T01:00:17.880127+0000","last_peered":"2024-09-22T01:00:17.880127+0000","last_clean":"2024-09-22T01:00:17.880127+0000","last_became_active":"2024-09-21T23:48:30.499944+0000","last_became_peered":"2024-09-21T23:48:30.499944+0000","last_unstale":"2024-09-22T01:00:17.880127+0000","last_undegraded":"2024-09-22T01:00:17.880127+0000","last_fullsized":"2024-09-22T01:00:17.880127+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:40:33.520270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016043909999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3815,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.878552+0000","last_change":"2024-09-21T23:48:32.522522+0000","last_active":"2024-09-22T01:00:17.878552+0000","last_peered":"2024-09-22T01:00:17.878552+0000","last_clean":"2024-09-22T01:00:17.878552+0000","last_became_active":"2024-09-21T23:48:30.498992+0000","last_became_peered":"2024-09-21T23:48:30.498992+0000","last_unstale":"2024-09-22T01:00:17.878552+0000","last_undegraded":"2024-09-22T01:00:17.878552+0000","last_fullsized":"2024-09-22T01:00:17.878552+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:30:17.385732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001283916,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3815,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.877569+0000","last_change":"2024-09-21T23:48:32.522399+0000","last_active":"2024-09-22T01:00:17.877569+0000","last_peered":"2024-09-22T01:00:17.877569+0000","last_clean":"2024-09-22T01:00:17.877569+0000","last_became_active":"2024-09-21T23:48:30.499210+0000","last_became_peered":"2024-09-21T23:48:30.499210+0000","last_unstale":"2024-09-22T01:00:17.877569+0000","last_undegraded":"2024-09-22T01:00:17.877569+0000","last_fullsized":"2024-09-22T01:00:17.877569+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:38:47.969858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012027369999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":3817,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:19.474795+0000","last_change":"2024-09-21T23:48:32.518542+0000","last_active":"2024-09-22T01:00:19.474795+0000","last_peered":"2024-09-22T01:00:19.474795+0000","last_clean":"2024-09-22T01:00:19.474795+0000","last_became_active":"2024-09-21T23:48:30.498643+0000","last_became_peered":"2024-09-21T23:48:30.498643+0000","last_unstale":"2024-09-22T01:00:19.474795+0000","last_undegraded":"2024-09-22T01:00:19.474795+0000","last_fullsized":"2024-09-22T01:00:19.474795+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:45:35.695733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019200619999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3815,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:19.363604+0000","last_change":"2024-09-21T23:48:32.513550+0000","last_active":"2024-09-22T01:00:19.363604+0000","last_peered":"2024-09-22T01:00:19.363604+0000","last_clean":"2024-09-22T01:00:19.363604+0000","last_became_active":"2024-09-21T23:48:30.500705+0000","last_became_peered":"2024-09-21T23:48:30.500705+0000","last_unstale":"2024-09-22T01:00:19.363604+0000","last_undegraded":"2024-09-22T01:00:19.363604+0000","last_fullsized":"2024-09-22T01:00:19.363604+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:21:01.132226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096276900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3817,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:19.364098+0000","last_change":"2024-09-21T23:48:32.513385+0000","last_active":"2024-09-22T01:00:19.364098+0000","last_peered":"2024-09-22T01:00:19.364098+0000","last_clean":"2024-09-22T01:00:19.364098+0000","last_became_active":"2024-09-21T23:48:30.501645+0000","last_became_peered":"2024-09-21T23:48:30.501645+0000","last_unstale":"2024-09-22T01:00:19.364098+0000","last_undegraded":"2024-09-22T01:00:19.364098+0000","last_fullsized":"2024-09-22T01:00:19.364098+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:12:39.920716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093488900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":3816,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.877374+0000","last_change":"2024-09-21T23:48:32.523316+0000","last_active":"2024-09-22T01:00:17.877374+0000","last_peered":"2024-09-22T01:00:17.877374+0000","last_clean":"2024-09-22T01:00:17.877374+0000","last_became_active":"2024-09-21T23:48:30.501010+0000","last_became_peered":"2024-09-21T23:48:30.501010+0000","last_unstale":"2024-09-22T01:00:17.877374+0000","last_undegraded":"2024-09-22T01:00:17.877374+0000","last_fullsized":"2024-09-22T01:00:17.877374+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:40:07.811058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001659017,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"556'125","reported_seq":3979,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.876769+0000","last_change":"2024-09-21T23:48:26.479535+0000","last_active":"2024-09-22T01:00:17.876769+0000","last_peered":"2024-09-22T01:00:17.876769+0000","last_clean":"2024-09-22T01:00:17.876769+0000","last_became_active":"2024-09-21T23:48:26.475455+0000","last_became_peered":"2024-09-21T23:48:26.475455+0000","last_unstale":"2024-09-22T01:00:17.876769+0000","last_undegraded":"2024-09-22T01:00:17.876769+0000","last_fullsized":"2024-09-22T01:00:17.876769+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:25.462486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:25.462486+0000","last_clean_scrub_stamp":"2024-09-21T23:48:25.462486+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T04:57:09.281011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739245,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1266920,"kb_used_data":4308,"kb_used_omap":1,"kb_used_meta":1262590,"kb_avail":93104920,"statfs":{"total":96636764160,"available":95339438080,"internally_reserved":0,"allocated":4411392,"data_stored":2424398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1292892613},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739245,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1191152,"kb_used_data":4892,"kb_used_omap":1,"kb_used_meta":1186238,"kb_avail":93180688,"statfs":{"total":96636764160,"available":95417024512,"internally_reserved":0,"allocated":5009408,"data_stored":3014785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1214708165},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739247,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1361904,"kb_used_data":4892,"kb_used_omap":1,"kb_used_meta":1356990,"kb_avail":93009936,"statfs":{"total":96636764160,"available":95242174464,"internally_reserved":0,"allocated":5009408,"data_stored":3014785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1389558214},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T01:01:08.487 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-22T01:01:08.487 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-22T01:01:08.792 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-22T01:01:08.792 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-22T01:01:08.811 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":4075,"stamp":"2024-09-22T01:01:08.112226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3819976,"kb_used_data":14092,"kb_used_omap":4,"kb_used_meta":3805819,"kb_avail":279295544,"statfs":{"total":289910292480,"available":285998637056,"internally_reserved":0,"allocated":14430208,"data_stored":8453968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":3897158992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002939"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3814,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.878316+0000","last_change":"2024-09-21T23:48:32.523721+0000","last_active":"2024-09-22T01:00:17.878316+0000","last_peered":"2024-09-22T01:00:17.878316+0000","last_clean":"2024-09-22T01:00:17.878316+0000","last_became_active":"2024-09-21T23:48:30.498942+0000","last_became_peered":"2024-09-21T23:48:30.498942+0000","last_unstale":"2024-09-22T01:00:17.878316+0000","last_undegraded":"2024-09-22T01:00:17.878316+0000","last_fullsized":"2024-09-22T01:00:17.878316+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:55:17.700760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019891520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3814,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.880127+0000","last_change":"2024-09-21T23:48:32.522974+0000","last_active":"2024-09-22T01:00:17.880127+0000","last_peered":"2024-09-22T01:00:17.880127+0000","last_clean":"2024-09-22T01:00:17.880127+0000","last_became_active":"2024-09-21T23:48:30.499944+0000","last_became_peered":"2024-09-21T23:48:30.499944+0000","last_unstale":"2024-09-22T01:00:17.880127+0000","last_undegraded":"2024-09-22T01:00:17.880127+0000","last_fullsized":"2024-09-22T01:00:17.880127+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:40:33.520270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016043909999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3815,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.878552+0000","last_change":"2024-09-21T23:48:32.522522+0000","last_active":"2024-09-22T01:00:17.878552+0000","last_peered":"2024-09-22T01:00:17.878552+0000","last_clean":"2024-09-22T01:00:17.878552+0000","last_became_active":"2024-09-21T23:48:30.498992+0000","last_became_peered":"2024-09-21T23:48:30.498992+0000","last_unstale":"2024-09-22T01:00:17.878552+0000","last_undegraded":"2024-09-22T01:00:17.878552+0000","last_fullsized":"2024-09-22T01:00:17.878552+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:30:17.385732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001283916,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3815,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.877569+0000","last_change":"2024-09-21T23:48:32.522399+0000","last_active":"2024-09-22T01:00:17.877569+0000","last_peered":"2024-09-22T01:00:17.877569+0000","last_clean":"2024-09-22T01:00:17.877569+0000","last_became_active":"2024-09-21T23:48:30.499210+0000","last_became_peered":"2024-09-21T23:48:30.499210+0000","last_unstale":"2024-09-22T01:00:17.877569+0000","last_undegraded":"2024-09-22T01:00:17.877569+0000","last_fullsized":"2024-09-22T01:00:17.877569+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:38:47.969858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012027369999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":3817,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:19.474795+0000","last_change":"2024-09-21T23:48:32.518542+0000","last_active":"2024-09-22T01:00:19.474795+0000","last_peered":"2024-09-22T01:00:19.474795+0000","last_clean":"2024-09-22T01:00:19.474795+0000","last_became_active":"2024-09-21T23:48:30.498643+0000","last_became_peered":"2024-09-21T23:48:30.498643+0000","last_unstale":"2024-09-22T01:00:19.474795+0000","last_undegraded":"2024-09-22T01:00:19.474795+0000","last_fullsized":"2024-09-22T01:00:19.474795+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:45:35.695733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019200619999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3815,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:19.363604+0000","last_change":"2024-09-21T23:48:32.513550+0000","last_active":"2024-09-22T01:00:19.363604+0000","last_peered":"2024-09-22T01:00:19.363604+0000","last_clean":"2024-09-22T01:00:19.363604+0000","last_became_active":"2024-09-21T23:48:30.500705+0000","last_became_peered":"2024-09-21T23:48:30.500705+0000","last_unstale":"2024-09-22T01:00:19.363604+0000","last_undegraded":"2024-09-22T01:00:19.363604+0000","last_fullsized":"2024-09-22T01:00:19.363604+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:21:01.132226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096276900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3817,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:19.364098+0000","last_change":"2024-09-21T23:48:32.513385+0000","last_active":"2024-09-22T01:00:19.364098+0000","last_peered":"2024-09-22T01:00:19.364098+0000","last_clean":"2024-09-22T01:00:19.364098+0000","last_became_active":"2024-09-21T23:48:30.501645+0000","last_became_peered":"2024-09-21T23:48:30.501645+0000","last_unstale":"2024-09-22T01:00:19.364098+0000","last_undegraded":"2024-09-22T01:00:19.364098+0000","last_fullsized":"2024-09-22T01:00:19.364098+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:12:39.920716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093488900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":3816,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.877374+0000","last_change":"2024-09-21T23:48:32.523316+0000","last_active":"2024-09-22T01:00:17.877374+0000","last_peered":"2024-09-22T01:00:17.877374+0000","last_clean":"2024-09-22T01:00:17.877374+0000","last_became_active":"2024-09-21T23:48:30.501010+0000","last_became_peered":"2024-09-21T23:48:30.501010+0000","last_unstale":"2024-09-22T01:00:17.877374+0000","last_undegraded":"2024-09-22T01:00:17.877374+0000","last_fullsized":"2024-09-22T01:00:17.877374+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:40:07.811058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001659017,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"556'125","reported_seq":3979,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.876769+0000","last_change":"2024-09-21T23:48:26.479535+0000","last_active":"2024-09-22T01:00:17.876769+0000","last_peered":"2024-09-22T01:00:17.876769+0000","last_clean":"2024-09-22T01:00:17.876769+0000","last_became_active":"2024-09-21T23:48:26.475455+0000","last_became_peered":"2024-09-21T23:48:26.475455+0000","last_unstale":"2024-09-22T01:00:17.876769+0000","last_undegraded":"2024-09-22T01:00:17.876769+0000","last_fullsized":"2024-09-22T01:00:17.876769+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:25.462486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:25.462486+0000","last_clean_scrub_stamp":"2024-09-21T23:48:25.462486+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T04:57:09.281011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739245,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1266920,"kb_used_data":4308,"kb_used_omap":1,"kb_used_meta":1262590,"kb_avail":93104920,"statfs":{"total":96636764160,"available":95339438080,"internally_reserved":0,"allocated":4411392,"data_stored":2424398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1292892613},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739245,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1191152,"kb_used_data":4892,"kb_used_omap":1,"kb_used_meta":1186238,"kb_avail":93180688,"statfs":{"total":96636764160,"available":95417024512,"internally_reserved":0,"allocated":5009408,"data_stored":3014785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1214708165},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739247,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1361904,"kb_used_data":4892,"kb_used_omap":1,"kb_used_meta":1356990,"kb_avail":93009936,"statfs":{"total":96636764160,"available":95242174464,"internally_reserved":0,"allocated":5009408,"data_stored":3014785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1389558214},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T01:01:08.812 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-22T01:01:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-22T01:01:09.120 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1896,"fsid":"432d7863-f767-472b-8b94-b82ff01e23c4","created":"2024-09-21T23:48:13.596459+0000","modified":"2024-09-22T01:00:17.861177+0000","last_up_change":"2024-09-21T23:48:24.461211+0000","last_in_change":"2024-09-21T23:48:14.548448+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T23:48:24.627658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T23:48:29.046255+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"517da62b-efb1-42ab-ae53-78dde46826da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":478776314},{"type":"v1","addr":"172.21.15.31:6817","nonce":478776314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":478776314},{"type":"v1","addr":"172.21.15.31:6819","nonce":478776314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":478776314},{"type":"v1","addr":"172.21.15.31:6823","nonce":478776314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":478776314},{"type":"v1","addr":"172.21.15.31:6821","nonce":478776314}]},"public_addr":"172.21.15.31:6817/478776314","cluster_addr":"172.21.15.31:6819/478776314","heartbeat_back_addr":"172.21.15.31:6823/478776314","heartbeat_front_addr":"172.21.15.31:6821/478776314","state":["exists","up"]},{"osd":1,"uuid":"4aa1c944-f2cb-4995-ba36-159cd9e3c1aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":437696049},{"type":"v1","addr":"172.21.15.31:6801","nonce":437696049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":437696049},{"type":"v1","addr":"172.21.15.31:6803","nonce":437696049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":437696049},{"type":"v1","addr":"172.21.15.31:6807","nonce":437696049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":437696049},{"type":"v1","addr":"172.21.15.31:6805","nonce":437696049}]},"public_addr":"172.21.15.31:6801/437696049","cluster_addr":"172.21.15.31:6803/437696049","heartbeat_back_addr":"172.21.15.31:6807/437696049","heartbeat_front_addr":"172.21.15.31:6805/437696049","state":["exists","up"]},{"osd":2,"uuid":"9eee3c58-3c7a-43f6-9684-6fecbcc505ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1610229797},{"type":"v1","addr":"172.21.15.31:6809","nonce":1610229797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1610229797},{"type":"v1","addr":"172.21.15.31:6811","nonce":1610229797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1610229797},{"type":"v1","addr":"172.21.15.31:6815","nonce":1610229797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1610229797},{"type":"v1","addr":"172.21.15.31:6813","nonce":1610229797}]},"public_addr":"172.21.15.31:6809/1610229797","cluster_addr":"172.21.15.31:6811/1610229797","heartbeat_back_addr":"172.21.15.31:6815/1610229797","heartbeat_front_addr":"172.21.15.31:6813/1610229797","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:21.292592+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:21.109724+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T23:48:21.056950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":3300,"length":1},{"begin":3311,"length":1},{"begin":3319,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T01:01:10.141 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-22T01:01:10.141 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-22T01:01:10.309 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-22T01:01:10.310 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-22T01:01:10.310 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-22T01:01:10.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-22T01:01:10.628 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 0 to deep-scrub 2024-09-22T01:01:10.649 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-22T01:01:10.649 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-22T01:01:10.802 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-22T01:01:10.803 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-22T01:01:10.803 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-22T01:01:10.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-22T01:01:11.108 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 1 to deep-scrub 2024-09-22T01:01:11.126 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-22T01:01:11.126 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-22T01:01:11.272 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-22T01:01:11.272 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-22T01:01:11.273 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-22T01:01:11.289 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-22T01:01:11.575 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 2 to deep-scrub 2024-09-22T01:01:11.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-22T01:01:11.883 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-22T01:01:11.883 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-22T01:01:11.902 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":4076,"stamp":"2024-09-22T01:01:10.112708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3819976,"kb_used_data":14092,"kb_used_omap":4,"kb_used_meta":3805819,"kb_avail":279295544,"statfs":{"total":289910292480,"available":285998637056,"internally_reserved":0,"allocated":14430208,"data_stored":8453968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":3897158992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002900"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3814,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.878316+0000","last_change":"2024-09-21T23:48:32.523721+0000","last_active":"2024-09-22T01:00:17.878316+0000","last_peered":"2024-09-22T01:00:17.878316+0000","last_clean":"2024-09-22T01:00:17.878316+0000","last_became_active":"2024-09-21T23:48:30.498942+0000","last_became_peered":"2024-09-21T23:48:30.498942+0000","last_unstale":"2024-09-22T01:00:17.878316+0000","last_undegraded":"2024-09-22T01:00:17.878316+0000","last_fullsized":"2024-09-22T01:00:17.878316+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:55:17.700760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019891520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3814,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.880127+0000","last_change":"2024-09-21T23:48:32.522974+0000","last_active":"2024-09-22T01:00:17.880127+0000","last_peered":"2024-09-22T01:00:17.880127+0000","last_clean":"2024-09-22T01:00:17.880127+0000","last_became_active":"2024-09-21T23:48:30.499944+0000","last_became_peered":"2024-09-21T23:48:30.499944+0000","last_unstale":"2024-09-22T01:00:17.880127+0000","last_undegraded":"2024-09-22T01:00:17.880127+0000","last_fullsized":"2024-09-22T01:00:17.880127+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:40:33.520270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016043909999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3815,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.878552+0000","last_change":"2024-09-21T23:48:32.522522+0000","last_active":"2024-09-22T01:00:17.878552+0000","last_peered":"2024-09-22T01:00:17.878552+0000","last_clean":"2024-09-22T01:00:17.878552+0000","last_became_active":"2024-09-21T23:48:30.498992+0000","last_became_peered":"2024-09-21T23:48:30.498992+0000","last_unstale":"2024-09-22T01:00:17.878552+0000","last_undegraded":"2024-09-22T01:00:17.878552+0000","last_fullsized":"2024-09-22T01:00:17.878552+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:30:17.385732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001283916,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3815,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.877569+0000","last_change":"2024-09-21T23:48:32.522399+0000","last_active":"2024-09-22T01:00:17.877569+0000","last_peered":"2024-09-22T01:00:17.877569+0000","last_clean":"2024-09-22T01:00:17.877569+0000","last_became_active":"2024-09-21T23:48:30.499210+0000","last_became_peered":"2024-09-21T23:48:30.499210+0000","last_unstale":"2024-09-22T01:00:17.877569+0000","last_undegraded":"2024-09-22T01:00:17.877569+0000","last_fullsized":"2024-09-22T01:00:17.877569+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:38:47.969858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012027369999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":3817,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:19.474795+0000","last_change":"2024-09-21T23:48:32.518542+0000","last_active":"2024-09-22T01:00:19.474795+0000","last_peered":"2024-09-22T01:00:19.474795+0000","last_clean":"2024-09-22T01:00:19.474795+0000","last_became_active":"2024-09-21T23:48:30.498643+0000","last_became_peered":"2024-09-21T23:48:30.498643+0000","last_unstale":"2024-09-22T01:00:19.474795+0000","last_undegraded":"2024-09-22T01:00:19.474795+0000","last_fullsized":"2024-09-22T01:00:19.474795+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:45:35.695733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019200619999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3815,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:19.363604+0000","last_change":"2024-09-21T23:48:32.513550+0000","last_active":"2024-09-22T01:00:19.363604+0000","last_peered":"2024-09-22T01:00:19.363604+0000","last_clean":"2024-09-22T01:00:19.363604+0000","last_became_active":"2024-09-21T23:48:30.500705+0000","last_became_peered":"2024-09-21T23:48:30.500705+0000","last_unstale":"2024-09-22T01:00:19.363604+0000","last_undegraded":"2024-09-22T01:00:19.363604+0000","last_fullsized":"2024-09-22T01:00:19.363604+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:21:01.132226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096276900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3817,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:19.364098+0000","last_change":"2024-09-21T23:48:32.513385+0000","last_active":"2024-09-22T01:00:19.364098+0000","last_peered":"2024-09-22T01:00:19.364098+0000","last_clean":"2024-09-22T01:00:19.364098+0000","last_became_active":"2024-09-21T23:48:30.501645+0000","last_became_peered":"2024-09-21T23:48:30.501645+0000","last_unstale":"2024-09-22T01:00:19.364098+0000","last_undegraded":"2024-09-22T01:00:19.364098+0000","last_fullsized":"2024-09-22T01:00:19.364098+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:12:39.920716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093488900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":3816,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.877374+0000","last_change":"2024-09-21T23:48:32.523316+0000","last_active":"2024-09-22T01:00:17.877374+0000","last_peered":"2024-09-22T01:00:17.877374+0000","last_clean":"2024-09-22T01:00:17.877374+0000","last_became_active":"2024-09-21T23:48:30.501010+0000","last_became_peered":"2024-09-21T23:48:30.501010+0000","last_unstale":"2024-09-22T01:00:17.877374+0000","last_undegraded":"2024-09-22T01:00:17.877374+0000","last_fullsized":"2024-09-22T01:00:17.877374+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:29.474124+0000","last_clean_scrub_stamp":"2024-09-21T23:48:29.474124+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:40:07.811058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001659017,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"556'125","reported_seq":3979,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:00:17.876769+0000","last_change":"2024-09-21T23:48:26.479535+0000","last_active":"2024-09-22T01:00:17.876769+0000","last_peered":"2024-09-22T01:00:17.876769+0000","last_clean":"2024-09-22T01:00:17.876769+0000","last_became_active":"2024-09-21T23:48:26.475455+0000","last_became_peered":"2024-09-21T23:48:26.475455+0000","last_unstale":"2024-09-22T01:00:17.876769+0000","last_undegraded":"2024-09-22T01:00:17.876769+0000","last_fullsized":"2024-09-22T01:00:17.876769+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T23:48:25.462486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T23:48:25.462486+0000","last_clean_scrub_stamp":"2024-09-21T23:48:25.462486+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T04:57:09.281011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739245,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1266920,"kb_used_data":4308,"kb_used_omap":1,"kb_used_meta":1262590,"kb_avail":93104920,"statfs":{"total":96636764160,"available":95339438080,"internally_reserved":0,"allocated":4411392,"data_stored":2424398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1292892613},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739246,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1191152,"kb_used_data":4892,"kb_used_omap":1,"kb_used_meta":1186238,"kb_avail":93180688,"statfs":{"total":96636764160,"available":95417024512,"internally_reserved":0,"allocated":5009408,"data_stored":3014785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1214708165},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739247,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1361904,"kb_used_data":4892,"kb_used_omap":1,"kb_used_meta":1356990,"kb_avail":93009936,"statfs":{"total":96636764160,"available":95242174464,"internally_reserved":0,"allocated":5009408,"data_stored":3014785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1389558214},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T01:01:11.903 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-21T23:48:29.474124+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=48, tm_sec=29, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=1, tm_min=1, tm_sec=9, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T01:01:11.903 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-21T23:48:29.474124+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=48, tm_sec=29, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=1, tm_min=1, tm_sec=9, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T01:01:11.903 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-21T23:48:29.474124+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=48, tm_sec=29, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=1, tm_min=1, tm_sec=9, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T01:01:11.903 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-21T23:48:29.474124+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=48, tm_sec=29, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=1, tm_min=1, tm_sec=9, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T01:01:11.903 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-21T23:48:29.474124+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=48, tm_sec=29, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=1, tm_min=1, tm_sec=9, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T01:01:11.903 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-21T23:48:29.474124+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=48, tm_sec=29, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=1, tm_min=1, tm_sec=9, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T01:01:11.903 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-21T23:48:29.474124+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=48, tm_sec=29, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=1, tm_min=1, tm_sec=9, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T01:01:11.903 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-21T23:48:29.474124+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=48, tm_sec=29, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=1, tm_min=1, tm_sec=9, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T01:01:11.904 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-21T23:48:25.462486+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=23, tm_min=48, tm_sec=25, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=1, tm_min=1, tm_sec=9, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T01:01:11.904 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-22T01:01:31.905 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-22T01:01:32.226 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-22T01:01:32.227 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-22T01:01:32.245 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":4087,"stamp":"2024-09-22T01:01:32.118256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3819964,"kb_used_data":14092,"kb_used_omap":4,"kb_used_meta":3805819,"kb_avail":279295556,"statfs":{"total":289910292480,"available":285998649344,"internally_reserved":0,"allocated":14430208,"data_stored":8453968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":3897158992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002984"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3823,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:01:16.792861+0000","last_change":"2024-09-22T01:01:16.792914+0000","last_active":"2024-09-22T01:01:16.792861+0000","last_peered":"2024-09-22T01:01:16.792861+0000","last_clean":"2024-09-22T01:01:16.792861+0000","last_became_active":"2024-09-21T23:48:30.498942+0000","last_became_peered":"2024-09-21T23:48:30.498942+0000","last_unstale":"2024-09-22T01:01:16.792861+0000","last_undegraded":"2024-09-22T01:01:16.792861+0000","last_fullsized":"2024-09-22T01:01:16.792861+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T01:01:16.792663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T01:01:16.792663+0000","last_clean_scrub_stamp":"2024-09-22T01:01:16.792663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:26:11.267092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019891520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3823,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:01:15.829031+0000","last_change":"2024-09-22T01:01:15.829084+0000","last_active":"2024-09-22T01:01:15.829031+0000","last_peered":"2024-09-22T01:01:15.829031+0000","last_clean":"2024-09-22T01:01:15.829031+0000","last_became_active":"2024-09-21T23:48:30.499944+0000","last_became_peered":"2024-09-21T23:48:30.499944+0000","last_unstale":"2024-09-22T01:01:15.829031+0000","last_undegraded":"2024-09-22T01:01:15.829031+0000","last_fullsized":"2024-09-22T01:01:15.829031+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T01:01:15.828829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T01:01:15.828829+0000","last_clean_scrub_stamp":"2024-09-22T01:01:15.828829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:09:29.620330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016043909999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3824,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:01:12.747917+0000","last_change":"2024-09-22T01:01:12.747973+0000","last_active":"2024-09-22T01:01:12.747917+0000","last_peered":"2024-09-22T01:01:12.747917+0000","last_clean":"2024-09-22T01:01:12.747917+0000","last_became_active":"2024-09-21T23:48:30.498992+0000","last_became_peered":"2024-09-21T23:48:30.498992+0000","last_unstale":"2024-09-22T01:01:12.747917+0000","last_undegraded":"2024-09-22T01:01:12.747917+0000","last_fullsized":"2024-09-22T01:01:12.747917+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T01:01:12.747706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T01:01:12.747706+0000","last_clean_scrub_stamp":"2024-09-22T01:01:12.747706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:34:11.954688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001283916,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3825,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:01:13.761447+0000","last_change":"2024-09-22T01:01:13.761447+0000","last_active":"2024-09-22T01:01:13.761447+0000","last_peered":"2024-09-22T01:01:13.761447+0000","last_clean":"2024-09-22T01:01:13.761447+0000","last_became_active":"2024-09-21T23:48:30.499210+0000","last_became_peered":"2024-09-21T23:48:30.499210+0000","last_unstale":"2024-09-22T01:01:13.761447+0000","last_undegraded":"2024-09-22T01:01:13.761447+0000","last_fullsized":"2024-09-22T01:01:13.761447+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T01:01:13.761175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T01:01:13.761175+0000","last_clean_scrub_stamp":"2024-09-22T01:01:13.761175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:33:26.500173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012027369999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"1896'3","reported_seq":3828,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:01:10.666382+0000","last_change":"2024-09-22T01:01:10.666382+0000","last_active":"2024-09-22T01:01:10.666382+0000","last_peered":"2024-09-22T01:01:10.666382+0000","last_clean":"2024-09-22T01:01:10.666382+0000","last_became_active":"2024-09-21T23:48:30.498643+0000","last_became_peered":"2024-09-21T23:48:30.498643+0000","last_unstale":"2024-09-22T01:01:10.666382+0000","last_undegraded":"2024-09-22T01:01:10.666382+0000","last_fullsized":"2024-09-22T01:01:10.666382+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"1896'3","last_scrub_stamp":"2024-09-22T01:01:10.666242+0000","last_deep_scrub":"1896'3","last_deep_scrub_stamp":"2024-09-22T01:01:10.666242+0000","last_clean_scrub_stamp":"2024-09-22T01:01:10.666242+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:08:13.336712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019200619999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3825,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:01:12.166181+0000","last_change":"2024-09-22T01:01:12.166181+0000","last_active":"2024-09-22T01:01:12.166181+0000","last_peered":"2024-09-22T01:01:12.166181+0000","last_clean":"2024-09-22T01:01:12.166181+0000","last_became_active":"2024-09-21T23:48:30.500705+0000","last_became_peered":"2024-09-21T23:48:30.500705+0000","last_unstale":"2024-09-22T01:01:12.166181+0000","last_undegraded":"2024-09-22T01:01:12.166181+0000","last_fullsized":"2024-09-22T01:01:12.166181+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T01:01:12.165924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T01:01:12.165924+0000","last_clean_scrub_stamp":"2024-09-22T01:01:12.165924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T10:11:02.116753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00096276900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3826,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:01:13.133148+0000","last_change":"2024-09-22T01:01:13.133204+0000","last_active":"2024-09-22T01:01:13.133148+0000","last_peered":"2024-09-22T01:01:13.133148+0000","last_clean":"2024-09-22T01:01:13.133148+0000","last_became_active":"2024-09-21T23:48:30.501645+0000","last_became_peered":"2024-09-21T23:48:30.501645+0000","last_unstale":"2024-09-22T01:01:13.133148+0000","last_undegraded":"2024-09-22T01:01:13.133148+0000","last_fullsized":"2024-09-22T01:01:13.133148+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T01:01:13.132944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T01:01:13.132944+0000","last_clean_scrub_stamp":"2024-09-22T01:01:13.132944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T13:00:09.999269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093488900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"1896'2","reported_seq":3827,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:01:14.793262+0000","last_change":"2024-09-22T01:01:14.793262+0000","last_active":"2024-09-22T01:01:14.793262+0000","last_peered":"2024-09-22T01:01:14.793262+0000","last_clean":"2024-09-22T01:01:14.793262+0000","last_became_active":"2024-09-21T23:48:30.501010+0000","last_became_peered":"2024-09-21T23:48:30.501010+0000","last_unstale":"2024-09-22T01:01:14.793262+0000","last_undegraded":"2024-09-22T01:01:14.793262+0000","last_fullsized":"2024-09-22T01:01:14.793262+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"1896'2","last_scrub_stamp":"2024-09-22T01:01:14.793007+0000","last_deep_scrub":"1896'2","last_deep_scrub_stamp":"2024-09-22T01:01:14.793007+0000","last_clean_scrub_stamp":"2024-09-22T01:01:14.793007+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:20:57.946626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001659017,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"1896'127","reported_seq":3990,"reported_epoch":1896,"state":"active+clean","last_fresh":"2024-09-22T01:01:11.783971+0000","last_change":"2024-09-22T01:01:11.784124+0000","last_active":"2024-09-22T01:01:11.783971+0000","last_peered":"2024-09-22T01:01:11.783971+0000","last_clean":"2024-09-22T01:01:11.783971+0000","last_became_active":"2024-09-21T23:48:26.475455+0000","last_became_peered":"2024-09-21T23:48:26.475455+0000","last_unstale":"2024-09-22T01:01:11.783971+0000","last_undegraded":"2024-09-22T01:01:11.783971+0000","last_fullsized":"2024-09-22T01:01:11.783971+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"1896'127","last_scrub_stamp":"2024-09-22T01:01:11.783744+0000","last_deep_scrub":"1896'127","last_deep_scrub_stamp":"2024-09-22T01:01:11.783744+0000","last_clean_scrub_stamp":"2024-09-22T01:01:11.783744+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:08:16.209467+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739250,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1266916,"kb_used_data":4308,"kb_used_omap":1,"kb_used_meta":1262590,"kb_avail":93104924,"statfs":{"total":96636764160,"available":95339442176,"internally_reserved":0,"allocated":4411392,"data_stored":2424398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1292892613},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739250,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1191148,"kb_used_data":4892,"kb_used_omap":1,"kb_used_meta":1186238,"kb_avail":93180692,"statfs":{"total":96636764160,"available":95417028608,"internally_reserved":0,"allocated":5009408,"data_stored":3014785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1214708165},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739252,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1361900,"kb_used_data":4892,"kb_used_omap":1,"kb_used_meta":1356990,"kb_avail":93009940,"statfs":{"total":96636764160,"available":95242178560,"internally_reserved":0,"allocated":5009408,"data_stored":3014785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1389558214},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T01:01:32.246 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-22T01:01:32.576 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-22T01:01:32.576 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-22T01:01:32.576 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-22T01:01:32.576 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T01:01:34.300 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 is failed for ~0s 2024-09-22T01:01:38.677 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-22T01:01:38.678 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-22T01:01:38.678 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T01:01:39.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 is failed for ~0s 2024-09-22T01:01:39.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 has been restored 2024-09-22T01:01:44.779 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-22T01:01:44.780 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-22T01:01:44.780 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T01:01:44.906 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 is failed for ~5s 2024-09-22T01:01:50.108 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 is failed for ~0s 2024-09-22T01:01:50.108 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 has been restored 2024-09-22T01:01:50.882 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-22T01:01:50.882 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-22T01:01:50.882 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-22T01:01:50.882 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T01:01:50.913 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-22T01:01:50.913 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-22T01:01:50.913 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-22T01:01:50.913 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T01:01:50.974 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-22T01:01:50.974 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-22T01:01:50.974 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-22T01:01:51.099 INFO:teuthology.orchestra.run.smithi031.stdout:2024-09-21T23:51:22.607862+0000 mon.a (mon.0) 248 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-22T01:01:51.100 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-09-22T01:01:51.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-22T01:01:51.117 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-22T01:01:51.176 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[WRN\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-22T01:01:51.235 INFO:teuthology.orchestra.run.smithi031.stdout:2024-09-21T23:51:22.607862+0000 mon.a (mon.0) 248 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-22T01:01:51.236 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-22T01:01:51.236 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-22T01:01:51.593 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-22T01:01:51.593 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-22T01:01:51.780 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-22T01:01:51.780 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-22T01:01:52.147 INFO:tasks.ceph:Archiving mon data... 2024-09-22T01:01:52.147 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914544/data/cluster1.mon.a.tgz 2024-09-22T01:01:52.147 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-09-22T01:01:52.162 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.rXYwoerv4N 2024-09-22T01:01:52.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.rXYwoerv4N 2024-09-22T01:01:52.361 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.rXYwoerv4N 2024-09-22T01:01:52.451 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.rXYwoerv4N is 546KB 2024-09-22T01:01:52.512 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.rXYwoerv4N 2024-09-22T01:01:52.518 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-22T01:01:52.518 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-22T01:01:52.564 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-22T01:01:52.634 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-22T01:01:52.663 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-22T01:01:52.663 INFO:tasks.ceph:Archiving crash dumps... 2024-09-22T01:01:52.664 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914544/remote/smithi031/crash 2024-09-22T01:01:52.665 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-22T01:01:52.691 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914544/remote/smithi060/crash 2024-09-22T01:01:52.692 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-22T01:01:52.717 INFO:tasks.ceph:Compressing logs... 2024-09-22T01:01:52.717 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-22T01:01:52.736 DEBUG:teuthology.orchestra.run.smithi060:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-22T01:01:52.753 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21518.log 2024-09-22T01:01:52.754 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25130.log 2024-09-22T01:01:52.755 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.tmp-client.admin.21518.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-09-22T01:01:52.755 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21518.log.gz 2024-09-22T01:01:52.755 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.25130.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25130.log.gz 2024-09-22T01:01:52.755 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30065.log 2024-09-22T01:01:52.755 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29832.log 2024-09-22T01:01:52.756 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.30065.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30065.log.gz 2024-09-22T01:01:52.756 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-09-22T01:01:52.757 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.29832.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29832.log.gz 2024-09-22T01:01:52.757 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41530.log 2024-09-22T01:01:52.757 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41402.log 2024-09-22T01:01:52.757 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.41530.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41530.log.gz 2024-09-22T01:01:52.757 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29702.log 2024-09-22T01:01:52.759 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.41402.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25375.log 2024-09-22T01:01:52.759 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41402.log.gz 2024-09-22T01:01:52.760 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.29702.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29702.log.gz 2024-09-22T01:01:52.760 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29911.log 2024-09-22T01:01:52.760 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/cluster1-client.admin.25375.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.29910.log 2024-09-22T01:01:52.760 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25375.log.gz 2024-09-22T01:01:52.760 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.29911.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29911.log.gz 2024-09-22T01:01:52.760 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-09-22T01:01:52.760 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.29910.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30059.log 2024-09-22T01:01:52.760 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29910.log.gz 2024-09-22T01:01:52.760 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29833.log 2024-09-22T01:01:52.763 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.30059.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-09-22T01:01:52.763 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30059.log.gz 2024-09-22T01:01:52.763 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.29833.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29833.log.gz 2024-09-22T01:01:52.763 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41324.log 2024-09-22T01:01:52.763 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29727.log 2024-09-22T01:01:52.763 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.41324.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41324.log.gz 2024-09-22T01:01:52.763 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29988.log 2024-09-22T01:01:52.763 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.29727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29773.log 2024-09-22T01:01:52.763 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.29988.log: 80.3% -- replaced with /var/log/ceph/cluster1-client.admin.29727.log.gz 2024-09-22T01:01:52.763 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-09-22T01:01:52.763 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29988.log.gz 2024-09-22T01:01:52.763 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.29773.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29773.log.gz 2024-09-22T01:01:52.764 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41480.log 2024-09-22T01:01:52.764 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25248.log 2024-09-22T01:01:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.41480.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41480.log.gz 2024-09-22T01:01:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-09-22T01:01:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.25248.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41272.log 2024-09-22T01:01:52.768 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25248.log.gz 2024-09-22T01:01:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29907.log 2024-09-22T01:01:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.41272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30163.log 2024-09-22T01:01:52.768 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41272.log.gz 2024-09-22T01:01:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.29907.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29907.log.gz 2024-09-22T01:01:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41428.log 2024-09-22T01:01:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.30163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41220.log 2024-09-22T01:01:52.768 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30163.log.gz 2024-09-22T01:01:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.41428.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41428.log.gz 2024-09-22T01:01:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41246.log 2024-09-22T01:01:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.41220.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41220.log.gz 2024-09-22T01:01:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25224.log 2024-09-22T01:01:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.41246.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/cluster1-client.admin.41246.log.gz 2024-09-22T01:01:52.769 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/cluster1-client.admin.30137.log 2024-09-22T01:01:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.25224.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25224.log.gz 2024-09-22T01:01:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28681.log 2024-09-22T01:01:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.30137.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41350.log 2024-09-22T01:01:52.769 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30137.log.gz 2024-09-22T01:01:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.28681.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28681.log.gz 2024-09-22T01:01:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25307.log 2024-09-22T01:01:52.773 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.41350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29981.log 2024-09-22T01:01:52.774 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41350.log.gz 2024-09-22T01:01:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.25307.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25307.log.gz 2024-09-22T01:01:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41506.log 2024-09-22T01:01:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.29981.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41376.log 2024-09-22T01:01:52.774 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29981.log.gz 2024-09-22T01:01:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.41506.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41506.log.gz 2024-09-22T01:01:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29747.log 2024-09-22T01:01:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.41376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29831.log 2024-09-22T01:01:52.774 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41376.log.gz 2024-09-22T01:01:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.29747.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29747.log.gz 2024-09-22T01:01:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41454.log 2024-09-22T01:01:52.774 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.29831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29989.log 2024-09-22T01:01:52.775 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29831.log.gz 2024-09-22T01:01:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.41454.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41454.log.gz 2024-09-22T01:01:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29677.log 2024-09-22T01:01:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.29989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29799.log 2024-09-22T01:01:52.775 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29989.log.gz 2024-09-22T01:01:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.29677.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29677.log.gz 2024-09-22T01:01:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-09-22T01:01:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.29799.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29799.log.gz 2024-09-22T01:01:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25272.log 2024-09-22T01:01:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30067.log 2024-09-22T01:01:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.25272.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25272.log.gz 2024-09-22T01:01:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30189.log 2024-09-22T01:01:52.775 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41298.log 2024-09-22T01:01:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.30067.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30067.log.gz 2024-09-22T01:01:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.30189.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30189.log.gz 2024-09-22T01:01:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-09-22T01:01:52.776 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.41298.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41298.log.gz 2024-09-22T01:01:52.781 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-osd.1.log: 87.3% -- replaced with /var/log/ceph/cluster1.log.gz 2024-09-22T01:01:52.787 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99312.log 2024-09-22T01:01:52.787 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131419.log 2024-09-22T01:01:52.788 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135461.log 2024-09-22T01:01:52.788 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.131419.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69606.log 2024-09-22T01:01:52.789 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91714.log 2024-09-22T01:01:52.789 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135461.log: /var/log/ceph/cluster1-client.mirror.69606.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42607.log 2024-09-22T01:01:52.790 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91714.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69357.log 2024-09-22T01:01:52.790 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77433.log 2024-09-22T01:01:52.791 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42607.log: /var/log/ceph/cluster1-client.mirror.69357.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85537.log 2024-09-22T01:01:52.791 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77433.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92722.log 2024-09-22T01:01:52.791 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.85537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107254.log 2024-09-22T01:01:52.792 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90330.log 2024-09-22T01:01:52.792 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49648.log 2024-09-22T01:01:52.793 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107254.log: /var/log/ceph/cluster1-client.mirror.90330.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78337.log 2024-09-22T01:01:52.794 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29691.log 2024-09-22T01:01:52.794 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.49648.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.49648.log.gz 2024-09-22T01:01:52.794 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78337.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36219.log 2024-09-22T01:01:52.794 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29691.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66733.log 2024-09-22T01:01:52.794 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29691.log.gz 2024-09-22T01:01:52.794 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90060.log 2024-09-22T01:01:52.797 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93310.log 2024-09-22T01:01:52.797 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66733.log: /var/log/ceph/cluster1-client.mirror.90060.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89218.log 2024-09-22T01:01:52.797 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93310.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135251.log 2024-09-22T01:01:52.797 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59101.log 2024-09-22T01:01:52.798 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119351.log 2024-09-22T01:01:52.798 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72946.log 2024-09-22T01:01:52.798 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.99312.log.gz 2024-09-22T01:01:52.798 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.119351.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130260.log 2024-09-22T01:01:52.798 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72946.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97946.log 2024-09-22T01:01:52.798 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.130260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67758.log 2024-09-22T01:01:52.798 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130260.log.gz 2024-09-22T01:01:52.798 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/cluster2-client.mirror.97946.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115560.log 2024-09-22T01:01:52.798 INFO:teuthology.orchestra.run.smithi060.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.85537.log.gz 2024-09-22T01:01:52.800 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67758.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42839.log 2024-09-22T01:01:52.800 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115560.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112032.log 2024-09-22T01:01:52.800 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.42839.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60025.log 2024-09-22T01:01:52.800 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112032.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116191.log 2024-09-22T01:01:52.800 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60025.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75784.log 2024-09-22T01:01:52.800 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116191.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57967.log 2024-09-22T01:01:52.800 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75784.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142765.log 2024-09-22T01:01:52.800 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29972.log 2024-09-22T01:01:52.802 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57967.log: /var/log/ceph/cluster2-client.admin.142765.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84906.log 2024-09-22T01:01:52.802 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142765.log.gz 2024-09-22T01:01:52.803 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.29972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66331.log 2024-09-22T01:01:52.803 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.84906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135209.log 2024-09-22T01:01:52.803 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66331.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85042.log 2024-09-22T01:01:52.803 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135209.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134791.log 2024-09-22T01:01:52.803 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.85042.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134874.log 2024-09-22T01:01:52.803 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44467.log 2024-09-22T01:01:52.803 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134874.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98996.log 2024-09-22T01:01:52.804 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44467.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69817.log 2024-09-22T01:01:52.804 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98996.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99800.log 2024-09-22T01:01:52.804 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69817.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136553.log 2024-09-22T01:01:52.805 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138616.log 2024-09-22T01:01:52.805 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136553.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98275.log 2024-09-22T01:01:52.805 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121944.log 2024-09-22T01:01:52.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137144.log 2024-09-22T01:01:52.807 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.75784.log.gz 2024-09-22T01:01:52.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83737.log 2024-09-22T01:01:52.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137144.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57884.log 2024-09-22T01:01:52.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83737.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69463.log 2024-09-22T01:01:52.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57884.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124792.log 2024-09-22T01:01:52.807 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69357.log.gz 2024-09-22T01:01:52.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69463.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132471.log 2024-09-22T01:01:52.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101572.log 2024-09-22T01:01:52.810 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126072.log 2024-09-22T01:01:52.810 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101572.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74651.log 2024-09-22T01:01:52.810 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110756.log 2024-09-22T01:01:52.810 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.74651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123237.log 2024-09-22T01:01:52.810 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110756.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142843.log 2024-09-22T01:01:52.810 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123237.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63271.log 2024-09-22T01:01:52.810 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.142843.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74537.log 2024-09-22T01:01:52.810 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142843.log.gz 2024-09-22T01:01:52.810 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.63271.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130849.log 2024-09-22T01:01:52.810 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42607.log.gz 2024-09-22T01:01:52.810 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.74537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134454.log 2024-09-22T01:01:52.812 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.130849.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51395.log 2024-09-22T01:01:52.812 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111780.log 2024-09-22T01:01:52.812 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42839.log.gz 2024-09-22T01:01:52.813 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51395.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55115.log 2024-09-22T01:01:52.813 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111780.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57842.log 2024-09-22T01:01:52.813 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55115.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92805.log 2024-09-22T01:01:52.813 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57842.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64183.log 2024-09-22T01:01:52.813 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135462.log 2024-09-22T01:01:52.813 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29716.log 2024-09-22T01:01:52.816 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135462.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65601.log 2024-09-22T01:01:52.816 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58597.log 2024-09-22T01:01:52.816 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29716.log.gz 2024-09-22T01:01:52.816 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79054.log 2024-09-22T01:01:52.816 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58597.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111507.log 2024-09-22T01:01:52.816 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61325.log 2024-09-22T01:01:52.816 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79054.log: /var/log/ceph/cluster1-client.mirror.111507.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99633.log 2024-09-22T01:01:52.816 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115100.log 2024-09-22T01:01:52.816 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99633.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132597.log 2024-09-22T01:01:52.816 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115100.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116903.log 2024-09-22T01:01:52.816 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132597.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62592.log 2024-09-22T01:01:52.816 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116903.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69018.log 2024-09-22T01:01:52.819 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116550.log 2024-09-22T01:01:52.819 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88943.log 2024-09-22T01:01:52.819 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116550.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124077.log 2024-09-22T01:01:52.819 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88943.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43086.log 2024-09-22T01:01:52.819 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124077.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87172.log 2024-09-22T01:01:52.819 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/cluster1-client.mirror.43086.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94069.log 2024-09-22T01:01:52.819 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.87172.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40356.log 2024-09-22T01:01:52.819 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94069.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60571.log 2024-09-22T01:01:52.819 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40356.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45960.log 2024-09-22T01:01:52.819 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99800.log.gz 2024-09-22T01:01:52.820 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60571.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83079.log 2024-09-22T01:01:52.821 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45960.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126131.log 2024-09-22T01:01:52.822 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83079.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57759.log 2024-09-22T01:01:52.822 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.83079.log.gz 2024-09-22T01:01:52.822 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96404.log 2024-09-22T01:01:52.822 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135714.log 2024-09-22T01:01:52.822 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101572.log.gz 2024-09-22T01:01:52.822 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96404.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58431.log 2024-09-22T01:01:52.822 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135714.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111718.log 2024-09-22T01:01:52.822 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141705.log 2024-09-22T01:01:52.822 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111718.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123972.log 2024-09-22T01:01:52.825 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/cluster1-client.mirror.141705.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80020.log 2024-09-22T01:01:52.825 INFO:teuthology.orchestra.run.smithi060.stderr: 68.3% -- replaced with /var/log/ceph/cluster1-client.mirror.141705.log.gz 2024-09-22T01:01:52.825 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72226.log 2024-09-22T01:01:52.825 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110756.log.gz 2024-09-22T01:01:52.825 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80020.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135799.log 2024-09-22T01:01:52.825 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.72226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114197.log 2024-09-22T01:01:52.825 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137986.log 2024-09-22T01:01:52.825 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114197.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75087.log 2024-09-22T01:01:52.825 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137986.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34871.log 2024-09-22T01:01:52.825 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111780.log.gz 2024-09-22T01:01:52.825 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.75087.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67255.log 2024-09-22T01:01:52.825 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34871.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100772.log 2024-09-22T01:01:52.825 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67255.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64461.log 2024-09-22T01:01:52.828 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67046.log 2024-09-22T01:01:52.828 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64461.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42643.log 2024-09-22T01:01:52.828 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67046.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111015.log 2024-09-22T01:01:52.828 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98996.log.gz 2024-09-22T01:01:52.828 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42643.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127079.log 2024-09-22T01:01:52.828 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111015.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64779.log 2024-09-22T01:01:52.828 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127079.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132091.log 2024-09-22T01:01:52.828 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64779.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138617.log 2024-09-22T01:01:52.829 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92513.log 2024-09-22T01:01:52.829 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97075.log 2024-09-22T01:01:52.829 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92513.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25371.log 2024-09-22T01:01:52.832 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.97075.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77715.log 2024-09-22T01:01:52.832 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.25371.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110520.log 2024-09-22T01:01:52.832 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.90330.log.gz 2024-09-22T01:01:52.832 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25371.log.gz 2024-09-22T01:01:52.832 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48085.log 2024-09-22T01:01:52.832 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.110520.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30262.log 2024-09-22T01:01:52.833 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38701.log 2024-09-22T01:01:52.833 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.30262.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134243.log 2024-09-22T01:01:52.833 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104322.log 2024-09-22T01:01:52.833 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38701.log: /var/log/ceph/cluster1-client.mirror.134243.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137019.log 2024-09-22T01:01:52.833 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88484.log 2024-09-22T01:01:52.833 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91714.log.gz 2024-09-22T01:01:52.833 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103837.log 2024-09-22T01:01:52.833 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137019.log: /var/log/ceph/cluster1-client.mirror.88484.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82751.log 2024-09-22T01:01:52.833 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103837.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133739.log 2024-09-22T01:01:52.833 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82751.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118017.log 2024-09-22T01:01:52.833 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.133739.log: /var/log/ceph/cluster1-client.mirror.82698.log 2024-09-22T01:01:52.838 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97510.log 2024-09-22T01:01:52.838 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118017.log: /var/log/ceph/cluster1-client.mirror.82698.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96643.log 2024-09-22T01:01:52.838 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.97510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127121.log 2024-09-22T01:01:52.838 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96643.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125965.log 2024-09-22T01:01:52.839 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101382.log 2024-09-22T01:01:52.839 INFO:teuthology.orchestra.run.smithi060.stderr: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.29972.log.gz 2024-09-22T01:01:52.839 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125965.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92429.log 2024-09-22T01:01:52.839 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117106.log 2024-09-22T01:01:52.839 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92429.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21600.log 2024-09-22T01:01:52.839 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.117106.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41156.log 2024-09-22T01:01:52.839 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph.tmp-client.admin.21600.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62467.log 2024-09-22T01:01:52.839 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21600.log.gz 2024-09-22T01:01:52.839 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47411.log 2024-09-22T01:01:52.839 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.62467.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121809.log 2024-09-22T01:01:52.839 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69606.log.gz 2024-09-22T01:01:52.839 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47411.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100141.log 2024-09-22T01:01:52.839 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121809.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74091.log 2024-09-22T01:01:52.839 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100141.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104257.log 2024-09-22T01:01:52.840 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.74091.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129753.log 2024-09-22T01:01:52.840 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104257.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91950.log 2024-09-22T01:01:52.840 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129753.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44467.log.gz 2024-09-22T01:01:52.840 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74786.log 2024-09-22T01:01:52.840 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91950.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139597.log 2024-09-22T01:01:52.840 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/cluster1-client.mirror.74786.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37601.log 2024-09-22T01:01:52.846 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139597.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91108.log 2024-09-22T01:01:52.846 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130212.log 2024-09-22T01:01:52.846 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91108.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127292.log 2024-09-22T01:01:52.846 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66331.log.gz 2024-09-22T01:01:52.846 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.130212.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63291.log 2024-09-22T01:01:52.847 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130212.log.gz 2024-09-22T01:01:52.847 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127292.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112306.log 2024-09-22T01:01:52.847 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.63291.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69774.log 2024-09-22T01:01:52.847 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101381.log 2024-09-22T01:01:52.847 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69774.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62698.log 2024-09-22T01:01:52.847 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101381.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41477.log 2024-09-22T01:01:52.847 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.62698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33168.log 2024-09-22T01:01:52.847 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109594.log 2024-09-22T01:01:52.847 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34747.log 2024-09-22T01:01:52.847 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.109594.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73290.log 2024-09-22T01:01:52.847 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34747.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54590.log 2024-09-22T01:01:52.847 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73290.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92722.log.gz 2024-09-22T01:01:52.847 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109467.log 2024-09-22T01:01:52.847 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54590.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75429.log 2024-09-22T01:01:52.848 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.109467.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139926.log 2024-09-22T01:01:52.848 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.75429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61719.log 2024-09-22T01:01:52.848 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139926.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78461.log 2024-09-22T01:01:52.848 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61719.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60572.log 2024-09-22T01:01:52.848 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78461.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118787.log 2024-09-22T01:01:52.848 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123237.log.gz 2024-09-22T01:01:52.848 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60572.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52673.log 2024-09-22T01:01:52.848 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111442.log 2024-09-22T01:01:52.848 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.52673.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119746.log 2024-09-22T01:01:52.848 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111442.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108199.log 2024-09-22T01:01:52.848 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.119746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66008.log 2024-09-22T01:01:52.856 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.108199.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29341.log 2024-09-22T01:01:52.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66008.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47831.log 2024-09-22T01:01:52.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53555.log 2024-09-22T01:01:52.857 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29341.log.gz 2024-09-22T01:01:52.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130434.log 2024-09-22T01:01:52.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53555.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89069.log 2024-09-22T01:01:52.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.130434.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48378.log 2024-09-22T01:01:52.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89069.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121857.log 2024-09-22T01:01:52.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48378.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37559.log 2024-09-22T01:01:52.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121857.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129116.log 2024-09-22T01:01:52.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37559.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121072.log 2024-09-22T01:01:52.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129116.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54256.log 2024-09-22T01:01:52.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89364.log 2024-09-22T01:01:52.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56036.log 2024-09-22T01:01:52.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89364.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82371.log 2024-09-22T01:01:52.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56036.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73508.log 2024-09-22T01:01:52.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82371.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126742.log 2024-09-22T01:01:52.858 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136553.log.gz 2024-09-22T01:01:52.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73508.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34327.log 2024-09-22T01:01:52.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126742.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74517.log 2024-09-22T01:01:52.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34327.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69143.log 2024-09-22T01:01:52.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.74517.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114239.log 2024-09-22T01:01:52.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116021.log 2024-09-22T01:01:52.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39324.log 2024-09-22T01:01:52.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69186.log 2024-09-22T01:01:52.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39324.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61231.log 2024-09-22T01:01:52.859 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135461.log.gz 2024-09-22T01:01:52.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69186.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34957.log 2024-09-22T01:01:52.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61231.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79264.log 2024-09-22T01:01:52.859 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135462.log.gz 2024-09-22T01:01:52.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34957.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83159.log 2024-09-22T01:01:52.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79264.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141335.log 2024-09-22T01:01:52.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127178.log 2024-09-22T01:01:52.859 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.83159.log.gz 2024-09-22T01:01:52.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.141335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110633.log 2024-09-22T01:01:52.859 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141335.log.gz 2024-09-22T01:01:52.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127178.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110757.log 2024-09-22T01:01:52.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110633.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142462.log 2024-09-22T01:01:52.860 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109132.log 2024-09-22T01:01:52.860 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.142462.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121983.log 2024-09-22T01:01:52.873 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65095.log 2024-09-22T01:01:52.874 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.142462.log.gz 2024-09-22T01:01:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121983.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106264.log 2024-09-22T01:01:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.65095.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37310.log 2024-09-22T01:01:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106264.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57548.log 2024-09-22T01:01:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37310.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99532.log 2024-09-22T01:01:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115039.log 2024-09-22T01:01:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89429.log 2024-09-22T01:01:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82582.log 2024-09-22T01:01:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115039.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.97946.log.gz 2024-09-22T01:01:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81618.log 2024-09-22T01:01:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46530.log 2024-09-22T01:01:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81618.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81237.log 2024-09-22T01:01:52.874 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46530.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78213.log 2024-09-22T01:01:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81237.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80525.log 2024-09-22T01:01:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78213.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123593.log 2024-09-22T01:01:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80525.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131517.log 2024-09-22T01:01:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123593.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92932.log 2024-09-22T01:01:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.131517.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115919.log 2024-09-22T01:01:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92932.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138784.log 2024-09-22T01:01:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112241.log 2024-09-22T01:01:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138784.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127056.log 2024-09-22T01:01:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112241.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74350.log 2024-09-22T01:01:52.875 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112032.log.gz 2024-09-22T01:01:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90182.log 2024-09-22T01:01:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.74350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47581.log 2024-09-22T01:01:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65703.log 2024-09-22T01:01:52.875 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47581.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54258.log 2024-09-22T01:01:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46991.log 2024-09-22T01:01:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33739.log 2024-09-22T01:01:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46991.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129768.log 2024-09-22T01:01:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42077.log 2024-09-22T01:01:52.876 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93310.log.gz 2024-09-22T01:01:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129768.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30023.log 2024-09-22T01:01:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42077.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41135.log 2024-09-22T01:01:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30023.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52760.log 2024-09-22T01:01:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129885.log 2024-09-22T01:01:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.52760.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137567.log 2024-09-22T01:01:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.129885.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142208.log 2024-09-22T01:01:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137567.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44886.log 2024-09-22T01:01:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.142208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41089.log 2024-09-22T01:01:52.876 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126848.log 2024-09-22T01:01:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.41089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58222.log 2024-09-22T01:01:52.877 INFO:teuthology.orchestra.run.smithi060.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-client.admin.142208.log.gz 2024-09-22T01:01:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126848.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95612.log 2024-09-22T01:01:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137777.log 2024-09-22T01:01:52.877 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124792.log.gz 2024-09-22T01:01:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.95612.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111890.log 2024-09-22T01:01:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137777.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133868.log 2024-09-22T01:01:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37989.log 2024-09-22T01:01:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111890.log: /var/log/ceph/cluster1-client.mirror.133868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94814.log 2024-09-22T01:01:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.37989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43962.log 2024-09-22T01:01:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79641.log 2024-09-22T01:01:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.94814.log: /var/log/ceph/cluster1-client.mirror.43962.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41384.log 2024-09-22T01:01:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79641.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141664.log 2024-09-22T01:01:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136092.log 2024-09-22T01:01:52.878 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89218.log.gz 2024-09-22T01:01:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.141664.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131305.log 2024-09-22T01:01:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136092.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64374.log 2024-09-22T01:01:52.878 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141664.log.gz 2024-09-22T01:01:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.131305.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130776.log 2024-09-22T01:01:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126597.log 2024-09-22T01:01:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.130776.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88942.log 2024-09-22T01:01:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126597.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134708.log 2024-09-22T01:01:52.878 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90060.log.gz 2024-09-22T01:01:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88942.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137354.log 2024-09-22T01:01:52.878 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134708.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142067.log 2024-09-22T01:01:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137354.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80776.log 2024-09-22T01:01:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.142067.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76418.log 2024-09-22T01:01:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36638.log 2024-09-22T01:01:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80776.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.142067.log.gz 2024-09-22T01:01:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125482.log 2024-09-22T01:01:52.891 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76418.log: /var/log/ceph/cluster1-client.mirror.36638.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73028.log 2024-09-22T01:01:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115307.log 2024-09-22T01:01:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.73028.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98023.log 2024-09-22T01:01:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71077.log 2024-09-22T01:01:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98023.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141980.log 2024-09-22T01:01:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71077.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49780.log 2024-09-22T01:01:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.141980.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117585.log 2024-09-22T01:01:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92679.log/var/log/ceph/cluster1-client.mirror.49780.log: 2024-09-22T01:01:52.892 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115560.log.gz 2024-09-22T01:01:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.117585.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68264.log 2024-09-22T01:01:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92679.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99039.log 2024-09-22T01:01:52.892 INFO:teuthology.orchestra.run.smithi060.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.141980.log.gz 2024-09-22T01:01:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68264.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141477.log 2024-09-22T01:01:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.99039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81996.log 2024-09-22T01:01:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.141477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134517.log 2024-09-22T01:01:52.893 INFO:teuthology.orchestra.run.smithi060.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.141477.log.gz 2024-09-22T01:01:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81996.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92051.log 2024-09-22T01:01:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134517.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91882.log 2024-09-22T01:01:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92051.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121157.log 2024-09-22T01:01:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102074.log 2024-09-22T01:01:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121157.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71352.log 2024-09-22T01:01:52.893 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77433.log.gz 2024-09-22T01:01:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102074.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118713.log 2024-09-22T01:01:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71352.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46907.log 2024-09-22T01:01:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.118713.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64652.log 2024-09-22T01:01:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46907.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75221.log 2024-09-22T01:01:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79473.log 2024-09-22T01:01:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64652.log: /var/log/ceph/cluster2-client.mirror.75221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112560.log 2024-09-22T01:01:52.893 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79473.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124452.log 2024-09-22T01:01:52.894 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67758.log.gz 2024-09-22T01:01:52.894 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112560.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127709.log 2024-09-22T01:01:52.894 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124452.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78169.log 2024-09-22T01:01:52.894 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127709.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57505.log 2024-09-22T01:01:52.894 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78169.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87216.log 2024-09-22T01:01:52.894 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/cluster1-client.mirror.57505.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134705.log 2024-09-22T01:01:52.894 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47687.log 2024-09-22T01:01:52.894 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134705.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137904.log 2024-09-22T01:01:52.894 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126240.log 2024-09-22T01:01:52.894 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134327.log 2024-09-22T01:01:52.894 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137904.log: /var/log/ceph/cluster1-client.mirror.126240.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79639.log 2024-09-22T01:01:52.894 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134327.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56791.log 2024-09-22T01:01:52.894 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79639.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95347.log 2024-09-22T01:01:52.894 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126746.log 2024-09-22T01:01:52.895 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56791.log: /var/log/ceph/cluster1-client.admin.95347.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41591.log 2024-09-22T01:01:52.895 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126746.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119766.log 2024-09-22T01:01:52.895 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/cluster2-client.mirror.41591.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30991.log 2024-09-22T01:01:52.895 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.119766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67632.log 2024-09-22T01:01:52.895 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30991.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65838.log 2024-09-22T01:01:52.895 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93919.log 2024-09-22T01:01:52.895 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67632.log: /var/log/ceph/cluster1-client.mirror.65838.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60025.log.gz 2024-09-22T01:01:52.895 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79219.log 2024-09-22T01:01:52.895 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136260.log 2024-09-22T01:01:52.895 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79219.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130507.log 2024-09-22T01:01:52.895 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100961.log 2024-09-22T01:01:52.895 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.130507.log: /var/log/ceph/cluster1-client.mirror.68622.log 2024-09-22T01:01:52.895 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100961.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82245.log 2024-09-22T01:01:52.895 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68622.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136679.log 2024-09-22T01:01:52.896 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132535.log/var/log/ceph/cluster1-client.mirror.82245.log: 2024-09-22T01:01:52.896 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136679.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84791.log 2024-09-22T01:01:52.896 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132535.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137608.log 2024-09-22T01:01:52.896 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80694.log 2024-09-22T01:01:52.896 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59101.log.gz 2024-09-22T01:01:52.896 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41990.log 2024-09-22T01:01:52.896 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80694.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141798.log 2024-09-22T01:01:52.896 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.41990.log: -- /var/log/ceph/cluster1-client.mirror.38088.log 2024-09-22T01:01:52.896 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.141798.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66307.log 2024-09-22T01:01:52.971 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124562.log 2024-09-22T01:01:52.972 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.38088.log: /var/log/ceph/cluster1-client.mirror.66307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120539.log 2024-09-22T01:01:52.972 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40357.log 2024-09-22T01:01:52.972 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124562.log: /var/log/ceph/cluster1-client.mirror.120539.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40773.log 2024-09-22T01:01:52.972 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/cluster2-client.admin.40357.log: -5 --verbose -- /var/log/ceph/cluster2-client.admin.118867.log 2024-09-22T01:01:52.972 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40773.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40773.log.gz 2024-09-22T01:01:52.972 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85972.log 2024-09-22T01:01:52.972 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118867.log: gzip 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78337.log.gz -5 2024-09-22T01:01:52.972 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.87067.log 2024-09-22T01:01:52.972 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.85972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99187.log 2024-09-22T01:01:52.972 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87067.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25254.log 2024-09-22T01:01:52.972 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.99187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44551.log 2024-09-22T01:01:52.972 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.25254.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77575.log 2024-09-22T01:01:52.973 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25254.log.gz 2024-09-22T01:01:52.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44551.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121878.log 2024-09-22T01:01:52.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77575.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39137.log 2024-09-22T01:01:52.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121878.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87003.log 2024-09-22T01:01:52.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39137.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72340.log 2024-09-22T01:01:52.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132572.log 2024-09-22T01:01:52.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.72340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30795.log 2024-09-22T01:01:52.973 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141798.log.gz 2024-09-22T01:01:52.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132572.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90832.log 2024-09-22T01:01:52.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30795.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137144.log.gz 2024-09-22T01:01:52.973 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124116.log 2024-09-22T01:01:52.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31792.log 2024-09-22T01:01:52.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124116.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60531.log 2024-09-22T01:01:52.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91252.log 2024-09-22T01:01:52.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60531.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35920.log 2024-09-22T01:01:52.974 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91252.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42790.log 2024-09-22T01:01:52.974 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35920.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74993.log 2024-09-22T01:01:52.974 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134454.log.gz 2024-09-22T01:01:52.974 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42790.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70581.log 2024-09-22T01:01:52.974 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.74993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113777.log 2024-09-22T01:01:52.974 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70581.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108562.log 2024-09-22T01:01:52.974 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113777.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69229.log 2024-09-22T01:01:52.974 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.108562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109674.log 2024-09-22T01:01:52.974 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69229.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41363.log 2024-09-22T01:01:52.974 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43561.log 2024-09-22T01:01:52.974 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41363.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78484.log 2024-09-22T01:01:52.974 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76755.log 2024-09-22T01:01:52.974 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78484.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128597.log 2024-09-22T01:01:52.974 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76755.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117590.log 2024-09-22T01:01:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128597.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123174.log 2024-09-22T01:01:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.117590.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30160.log 2024-09-22T01:01:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123174.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51673.log 2024-09-22T01:01:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94548.log 2024-09-22T01:01:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.30160.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126072.log.gz 2024-09-22T01:01:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51673.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68682.log 2024-09-22T01:01:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.94548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72178.log 2024-09-22T01:01:52.975 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94548.log.gz 2024-09-22T01:01:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68682.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39359.log 2024-09-22T01:01:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.72178.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100938.log 2024-09-22T01:01:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39359.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46868.log 2024-09-22T01:01:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100938.log: gzip -5 --verbose -- 90.4% /var/log/ceph/cluster1-client.mirror.56203.log 2024-09-22T01:01:52.975 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.126131.log.gz 2024-09-22T01:01:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61486.log 2024-09-22T01:01:52.976 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56203.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100960.log 2024-09-22T01:01:52.976 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/cluster1-client.admin.61486.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88177.log 2024-09-22T01:01:52.976 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100960.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79896.log 2024-09-22T01:01:52.976 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88177.log: gzip -5 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116191.log.gz 2024-09-22T01:01:52.976 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.28984.log 2024-09-22T01:01:52.976 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79896.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79895.log 2024-09-22T01:01:52.976 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.28984.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28984.log.gz 2024-09-22T01:01:52.976 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123973.log 2024-09-22T01:01:52.976 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79895.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64203.log 2024-09-22T01:01:52.976 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123973.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70722.log 2024-09-22T01:01:52.976 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/cluster2-client.mirror.64203.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.75376.log 2024-09-22T01:01:52.976 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111356.log 2024-09-22T01:01:52.976 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.75376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88529.log 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71077.log.gz 2024-09-22T01:01:52.977 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:01:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111356.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133658.log 2024-09-22T01:01:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88529.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64990.log 2024-09-22T01:01:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133658.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90269.log 2024-09-22T01:01:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64990.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48546.log 2024-09-22T01:01:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112097.log 2024-09-22T01:01:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48546.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126702.log 2024-09-22T01:01:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112097.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66733.log.gz 2024-09-22T01:01:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89848.log 2024-09-22T01:01:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127017.log 2024-09-22T01:01:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89848.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71287.log 2024-09-22T01:01:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127017.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119225.log 2024-09-22T01:01:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71287.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76374.log 2024-09-22T01:01:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.119225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116234.log 2024-09-22T01:01:52.978 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119225.log.gz 2024-09-22T01:01:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.76374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38287.log 2024-09-22T01:01:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116234.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49499.log 2024-09-22T01:01:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.38287.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141752.log 2024-09-22T01:01:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49499.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107645.log 2024-09-22T01:01:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.141752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67673.log 2024-09-22T01:01:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.107645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71354.log 2024-09-22T01:01:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67673.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108863.log 2024-09-22T01:01:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71354.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129111.log 2024-09-22T01:01:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.108863.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118015.log 2024-09-22T01:01:52.978 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.141752.log.gz 2024-09-22T01:01:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119518.log 2024-09-22T01:01:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118015.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98702.log 2024-09-22T01:01:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.119518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114451.log 2024-09-22T01:01:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71898.log 2024-09-22T01:01:52.979 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69463.log.gz 2024-09-22T01:01:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114451.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42710.log 2024-09-22T01:01:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45897.log 2024-09-22T01:01:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.42710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76522.log 2024-09-22T01:01:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45897.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58975.log 2024-09-22T01:01:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.76522.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84944.log 2024-09-22T01:01:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58975.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122039.log 2024-09-22T01:01:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.84944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122753.log 2024-09-22T01:01:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70068.log 2024-09-22T01:01:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113754.log 2024-09-22T01:01:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124287.log 2024-09-22T01:01:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113754.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55532.log 2024-09-22T01:01:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124287.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135881.log 2024-09-22T01:01:52.980 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121944.log.gz 2024-09-22T01:01:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55532.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29215.log 2024-09-22T01:01:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47453.log 2024-09-22T01:01:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36823.log 2024-09-22T01:01:52.980 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29215.log.gz 2024-09-22T01:01:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47453.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118786.log 2024-09-22T01:01:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36823.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49247.log 2024-09-22T01:01:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118786.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121008.log 2024-09-22T01:01:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49247.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56332.log 2024-09-22T01:01:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121008.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120316.log 2024-09-22T01:01:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48545.log 2024-09-22T01:01:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.120316.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72200.log 2024-09-22T01:01:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82036.log 2024-09-22T01:01:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136561.log 2024-09-22T01:01:52.980 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.72200.log.gz 2024-09-22T01:01:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82036.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112199.log 2024-09-22T01:01:52.981 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100141.log.gz 2024-09-22T01:01:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59377.log 2024-09-22T01:01:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112199.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102702.log 2024-09-22T01:01:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59377.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98703.log 2024-09-22T01:01:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36763.log 2024-09-22T01:01:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71076.log 2024-09-22T01:01:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36763.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105222.log 2024-09-22T01:01:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71076.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36113.log 2024-09-22T01:01:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105222.log: /var/log/ceph/cluster1-client.mirror.36113.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69398.log 2024-09-22T01:01:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30902.log 2024-09-22T01:01:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105504.log 2024-09-22T01:01:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49370.log 2024-09-22T01:01:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105504.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98849.log 2024-09-22T01:01:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49370.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63405.log 2024-09-22T01:01:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98849.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117941.log 2024-09-22T01:01:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.63405.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68178.log 2024-09-22T01:01:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.117941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125692.log 2024-09-22T01:01:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68178.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72114.log 2024-09-22T01:01:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31643.log 2024-09-22T01:01:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.72114.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43154.log 2024-09-22T01:01:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31643.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89111.log 2024-09-22T01:01:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.43154.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107440.log 2024-09-22T01:01:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89111.log: gzip 90.4% -5 --verbose -- -- replaced with /var/log/ceph/cluster1-client.mirror.92932.log.gz /var/log/ceph/cluster1-client.mirror.89849.log 2024-09-22T01:01:52.982 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:01:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107440.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58850.log 2024-09-22T01:01:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89849.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112453.log 2024-09-22T01:01:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58135.log 2024-09-22T01:01:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112980.log 2024-09-22T01:01:52.983 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25347.log 2024-09-22T01:01:52.983 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89364.log.gz 2024-09-22T01:01:52.983 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98491.log 2024-09-22T01:01:52.983 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112980.log: /var/log/ceph/cluster2-client.admin.25347.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142997.log 2024-09-22T01:01:52.983 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25347.log.gz 2024-09-22T01:01:52.983 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62251.log 2024-09-22T01:01:52.983 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.142997.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48672.log 2024-09-22T01:01:52.983 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142997.log.gz 2024-09-22T01:01:52.983 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114343.log 2024-09-22T01:01:52.983 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48672.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103060.log 2024-09-22T01:01:52.983 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114343.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138743.log 2024-09-22T01:01:52.983 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103060.log: /var/log/ceph/cluster1-client.mirror.138743.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.92805.log.gz 2024-09-22T01:01:52.983 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46865.log 2024-09-22T01:01:52.984 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137483.log 2024-09-22T01:01:52.984 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46865.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90959.log 2024-09-22T01:01:52.984 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93419.log 2024-09-22T01:01:52.984 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90959.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136133.log 2024-09-22T01:01:52.984 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93419.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56246.log 2024-09-22T01:01:52.984 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136133.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122390.log 2024-09-22T01:01:52.984 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56246.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75489.log 2024-09-22T01:01:52.984 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37453.log 2024-09-22T01:01:52.984 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75489.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138407.log 2024-09-22T01:01:52.984 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134582.log 2024-09-22T01:01:52.984 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138407.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77536.log 2024-09-22T01:01:52.984 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55826.log 2024-09-22T01:01:52.984 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111718.log.gz 2024-09-22T01:01:52.984 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83352.log 2024-09-22T01:01:52.984 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55826.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109379.log 2024-09-22T01:01:52.984 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83352.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38788.log 2024-09-22T01:01:52.985 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.83352.log.gz 2024-09-22T01:01:52.985 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131346.log 2024-09-22T01:01:52.985 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38788.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39613.log 2024-09-22T01:01:52.985 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.131346.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94176.log 2024-09-22T01:01:52.985 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39613.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133410.log 2024-09-22T01:01:52.985 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94176.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104384.log 2024-09-22T01:01:52.985 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116234.log.gz 2024-09-22T01:01:52.985 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133410.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138533.log 2024-09-22T01:01:52.985 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136007.log 2024-09-22T01:01:52.985 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138533.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91672.log 2024-09-22T01:01:52.985 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136007.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100058.log 2024-09-22T01:01:52.985 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91672.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105783.log 2024-09-22T01:01:52.985 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100058.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113925.log 2024-09-22T01:01:52.985 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.105783.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74338.log 2024-09-22T01:01:52.985 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105783.log.gz 2024-09-22T01:01:52.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113925.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123090.log 2024-09-22T01:01:52.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.74338.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33400.log 2024-09-22T01:01:52.986 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74338.log.gz 2024-09-22T01:01:52.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123090.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74227.log 2024-09-22T01:01:52.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101066.log 2024-09-22T01:01:52.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.74227.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100226.log 2024-09-22T01:01:52.986 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74227.log.gz 2024-09-22T01:01:52.986 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111507.log.gz 2024-09-22T01:01:52.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101066.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97893.log 2024-09-22T01:01:52.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35039.log 2024-09-22T01:01:52.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.97893.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118026.log 2024-09-22T01:01:52.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91529.log 2024-09-22T01:01:52.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136136.log 2024-09-22T01:01:52.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91529.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61677.log 2024-09-22T01:01:52.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123486.log 2024-09-22T01:01:52.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61677.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40350.log 2024-09-22T01:01:52.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123486.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60070.log 2024-09-22T01:01:52.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76012.log 2024-09-22T01:01:52.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60070.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126913.log 2024-09-22T01:01:52.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120263.log 2024-09-22T01:01:52.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126913.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126387.log 2024-09-22T01:01:52.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.120263.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110185.log 2024-09-22T01:01:52.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126387.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110229.log 2024-09-22T01:01:52.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.110185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103039.log 2024-09-22T01:01:52.987 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123551.log 2024-09-22T01:01:52.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110229.log: /var/log/ceph/cluster1-client.mirror.103039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88176.log 2024-09-22T01:01:52.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123551.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31767.log 2024-09-22T01:01:52.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88176.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39762.log 2024-09-22T01:01:52.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31767.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97920.log 2024-09-22T01:01:52.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39762.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44301.log 2024-09-22T01:01:52.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.97920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32049.log 2024-09-22T01:01:52.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44301.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82791.log 2024-09-22T01:01:52.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32049.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130414.log 2024-09-22T01:01:52.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82791.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118161.log 2024-09-22T01:01:52.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.130414.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70026.log 2024-09-22T01:01:52.988 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127121.log.gz 2024-09-22T01:01:52.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118161.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129767.log 2024-09-22T01:01:52.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75526.log 2024-09-22T01:01:52.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129767.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124791.log 2024-09-22T01:01:52.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75526.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108427.log 2024-09-22T01:01:52.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108676.log 2024-09-22T01:01:52.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.108427.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77794.log 2024-09-22T01:01:52.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.108676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135294.log 2024-09-22T01:01:52.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77794.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122922.log 2024-09-22T01:01:52.989 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124077.log.gz 2024-09-22T01:01:52.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135294.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134517.log.gz 2024-09-22T01:01:52.989 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71243.log 2024-09-22T01:01:52.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122922.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34536.log 2024-09-22T01:01:52.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71243.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64716.log 2024-09-22T01:01:52.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113291.log 2024-09-22T01:01:52.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81198.log 2024-09-22T01:01:52.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113291.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41290.log 2024-09-22T01:01:52.990 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81198.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115920.log 2024-09-22T01:01:52.990 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.41290.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33001.log 2024-09-22T01:01:52.990 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91736.log 2024-09-22T01:01:52.990 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33001.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40906.log 2024-09-22T01:01:52.990 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91736.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30729.log 2024-09-22T01:01:52.990 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135209.log.gz 2024-09-22T01:01:52.990 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56875.log 2024-09-22T01:01:52.990 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30729.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65047.log 2024-09-22T01:01:52.990 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92579.log 2024-09-22T01:01:52.990 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47537.log 2024-09-22T01:01:52.990 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65047.log: /var/log/ceph/cluster1-client.mirror.92579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93963.log 2024-09-22T01:01:52.990 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135251.log.gz 2024-09-22T01:01:52.990 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36613.log 2024-09-22T01:01:52.990 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47537.log: /var/log/ceph/cluster1-client.mirror.93963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102577.log 2024-09-22T01:01:52.990 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36613.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68556.log 2024-09-22T01:01:52.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102577.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130061.log 2024-09-22T01:01:52.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68556.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106778.log 2024-09-22T01:01:52.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.130061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61954.log 2024-09-22T01:01:52.991 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25323.log 2024-09-22T01:01:52.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.106778.log: /var/log/ceph/cluster1-client.admin.61954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122480.log 2024-09-22T01:01:52.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.25323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30927.log 2024-09-22T01:01:52.991 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25323.log.gz 2024-09-22T01:01:52.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122480.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81952.log 2024-09-22T01:01:52.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30927.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47244.log 2024-09-22T01:01:52.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81952.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81365.log 2024-09-22T01:01:52.991 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110633.log.gz 2024-09-22T01:01:52.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47244.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32269.log 2024-09-22T01:01:52.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37991.log 2024-09-22T01:01:52.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104802.log 2024-09-22T01:01:52.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.37991.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131601.log 2024-09-22T01:01:52.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47957.log 2024-09-22T01:01:52.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75594.log 2024-09-22T01:01:52.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47957.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114029.log 2024-09-22T01:01:52.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75594.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39712.log 2024-09-22T01:01:52.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112410.log 2024-09-22T01:01:52.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121302.log 2024-09-22T01:01:52.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112410.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132424.log 2024-09-22T01:01:52.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121302.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74491.log 2024-09-22T01:01:52.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132424.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46655.log 2024-09-22T01:01:52.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.74491.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46571.log 2024-09-22T01:01:52.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120149.log 2024-09-22T01:01:52.992 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134791.log.gz 2024-09-22T01:01:52.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46571.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122814.log 2024-09-22T01:01:52.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.120149.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137102.log 2024-09-22T01:01:52.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113882.log 2024-09-22T01:01:52.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137102.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87105.log 2024-09-22T01:01:52.993 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134874.log.gz 2024-09-22T01:01:52.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115712.log 2024-09-22T01:01:52.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87105.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82539.log 2024-09-22T01:01:52.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77436.log 2024-09-22T01:01:52.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82539.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48461.log 2024-09-22T01:01:52.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77436.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87871.log 2024-09-22T01:01:52.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48461.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35038.log 2024-09-22T01:01:52.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87871.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94296.log 2024-09-22T01:01:52.993 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46211.log 2024-09-22T01:01:52.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35038.log: /var/log/ceph/cluster1-client.admin.94296.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49143.log 2024-09-22T01:01:52.993 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94296.log.gz 2024-09-22T01:01:52.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46211.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89488.log 2024-09-22T01:01:52.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55616.log 2024-09-22T01:01:52.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89488.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78631.log 2024-09-22T01:01:52.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81744.log 2024-09-22T01:01:52.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78631.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92430.log 2024-09-22T01:01:52.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81744.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40353.log 2024-09-22T01:01:52.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125062.log 2024-09-22T01:01:52.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102913.log 2024-09-22T01:01:52.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125062.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64317.log 2024-09-22T01:01:52.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107508.log 2024-09-22T01:01:52.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64317.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113156.log 2024-09-22T01:01:52.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107508.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96249.log 2024-09-22T01:01:53.078 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128915.log 2024-09-22T01:01:53.078 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113156.log: /var/log/ceph/cluster1-client.admin.96249.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121556.log 2024-09-22T01:01:53.078 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.admin.128915.log: -- /var/log/ceph/cluster1-client.admin.83620.log 2024-09-22T01:01:53.078 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.121556.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34662.log 2024-09-22T01:01:53.078 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83620.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121489.log 2024-09-22T01:01:53.078 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34662.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81659.log 2024-09-22T01:01:53.078 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121489.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63929.log 2024-09-22T01:01:53.078 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114929.log 2024-09-22T01:01:53.078 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122753.log.gz 2024-09-22T01:01:53.079 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.63929.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109720.log 2024-09-22T01:01:53.079 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114929.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120863.log 2024-09-22T01:01:53.079 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104592.log 2024-09-22T01:01:53.079 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120863.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142895.log 2024-09-22T01:01:53.079 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112494.log 2024-09-22T01:01:53.079 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.142895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53961.log 2024-09-22T01:01:53.079 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142895.log.gz 2024-09-22T01:01:53.079 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112494.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101571.log 2024-09-22T01:01:53.079 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53961.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51253.log 2024-09-22T01:01:53.079 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101571.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140908.log 2024-09-22T01:01:53.079 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.51253.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66535.log 2024-09-22T01:01:53.079 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108924.log 2024-09-22T01:01:53.079 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66535.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79010.log 2024-09-22T01:01:53.079 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.108924.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124497.log 2024-09-22T01:01:53.080 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79010.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96577.log 2024-09-22T01:01:53.107 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124497.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89302.log 2024-09-22T01:01:53.107 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75489.log.gz 2024-09-22T01:01:53.107 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.96577.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.75526.log.gz 2024-09-22T01:01:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62259.log 2024-09-22T01:01:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118384.log 2024-09-22T01:01:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89302.log: /var/log/ceph/cluster2-client.admin.62259.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63206.log 2024-09-22T01:01:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93771.log 2024-09-22T01:01:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.63206.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43300.log 2024-09-22T01:01:53.108 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63206.log.gz 2024-09-22T01:01:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49541.log 2024-09-22T01:01:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43300.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121451.log 2024-09-22T01:01:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49541.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73781.log 2024-09-22T01:01:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121451.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62042.log 2024-09-22T01:01:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.73781.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89614.log 2024-09-22T01:01:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62042.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105269.log 2024-09-22T01:01:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89614.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87594.log 2024-09-22T01:01:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43484.log 2024-09-22T01:01:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87594.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88791.log 2024-09-22T01:01:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43484.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74973.log 2024-09-22T01:01:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88791.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106774.log 2024-09-22T01:01:53.109 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69817.log.gz 2024-09-22T01:01:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.74973.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31029.log 2024-09-22T01:01:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.106774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87695.log 2024-09-22T01:01:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59773.log 2024-09-22T01:01:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87695.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123889.log 2024-09-22T01:01:53.109 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63271.log.gz 2024-09-22T01:01:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59773.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117035.log 2024-09-22T01:01:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123889.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108495.log 2024-09-22T01:01:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.117035.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88615.log 2024-09-22T01:01:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.108495.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89219.log 2024-09-22T01:01:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88615.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119807.log 2024-09-22T01:01:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114450.log 2024-09-22T01:01:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.119807.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89949.log 2024-09-22T01:01:53.110 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114450.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58011.log 2024-09-22T01:01:53.110 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38696.log 2024-09-22T01:01:53.110 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69314.log 2024-09-22T01:01:53.110 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58011.log: /var/log/ceph/cluster1-client.admin.38696.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71413.log 2024-09-22T01:01:53.110 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82185.log 2024-09-22T01:01:53.110 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71413.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84504.log 2024-09-22T01:01:53.110 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125651.log 2024-09-22T01:01:53.110 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/cluster2-client.mirror.84504.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71813.log 2024-09-22T01:01:53.110 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134327.log.gz 2024-09-22T01:01:53.110 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80273.log 2024-09-22T01:01:53.110 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/cluster1-client.mirror.71813.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34450.log 2024-09-22T01:01:53.110 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47372.log 2024-09-22T01:01:53.110 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34450.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73950.log 2024-09-22T01:01:53.110 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47372.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72039.log 2024-09-22T01:01:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84279.log 2024-09-22T01:01:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111654.log 2024-09-22T01:01:53.111 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.72039.log.gz 2024-09-22T01:01:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84279.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32090.log 2024-09-22T01:01:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119921.log 2024-09-22T01:01:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111654.log: /var/log/ceph/cluster1-client.mirror.32090.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79220.log 2024-09-22T01:01:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.119921.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49847.log 2024-09-22T01:01:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65964.log 2024-09-22T01:01:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49847.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139354.log 2024-09-22T01:01:53.111 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138616.log.gz 2024-09-22T01:01:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65964.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64753.log 2024-09-22T01:01:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139354.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104466.log 2024-09-22T01:01:53.111 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.139354.log.gz 2024-09-22T01:01:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58891.log 2024-09-22T01:01:53.112 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138617.log.gz 2024-09-22T01:01:53.112 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104466.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69396.log 2024-09-22T01:01:53.112 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58891.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101573.log 2024-09-22T01:01:53.112 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103376.log 2024-09-22T01:01:53.112 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101573.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40117.log 2024-09-22T01:01:53.113 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103376.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62040.log 2024-09-22T01:01:53.113 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40117.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47076.log 2024-09-22T01:01:53.113 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71412.log 2024-09-22T01:01:53.113 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62040.log: /var/log/ceph/cluster1-client.mirror.47076.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113611.log 2024-09-22T01:01:53.114 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71412.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124855.log 2024-09-22T01:01:53.114 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111570.log 2024-09-22T01:01:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124855.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103270.log 2024-09-22T01:01:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111570.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61498.log 2024-09-22T01:01:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35270.log 2024-09-22T01:01:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63473.log 2024-09-22T01:01:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53492.log 2024-09-22T01:01:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.63473.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37243.log 2024-09-22T01:01:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135083.log 2024-09-22T01:01:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139096.log 2024-09-22T01:01:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37243.log: /var/log/ceph/cluster1-client.mirror.135083.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132895.log 2024-09-22T01:01:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36258.log 2024-09-22T01:01:53.115 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.139096.log.gz 2024-09-22T01:01:53.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132895.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62051.log 2024-09-22T01:01:53.117 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.33168.log.gz 2024-09-22T01:01:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38926.log 2024-09-22T01:01:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62051.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115668.log 2024-09-22T01:01:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38926.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79429.log 2024-09-22T01:01:53.117 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115668.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135127.log 2024-09-22T01:01:53.118 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79429.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140182.log 2024-09-22T01:01:53.118 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135127.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125232.log 2024-09-22T01:01:53.118 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88109.log 2024-09-22T01:01:53.118 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43086.log.gz 2024-09-22T01:01:53.118 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125232.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68430.log 2024-09-22T01:01:53.118 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88109.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115459.log 2024-09-22T01:01:53.118 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41612.log 2024-09-22T01:01:53.118 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138029.log 2024-09-22T01:01:53.118 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115459.log: /var/log/ceph/cluster1-client.mirror.41612.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65904.log 2024-09-22T01:01:53.118 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71476.log 2024-09-22T01:01:53.118 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79095.log 2024-09-22T01:01:53.118 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101946.log 2024-09-22T01:01:53.118 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79095.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67993.log 2024-09-22T01:01:53.118 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101946.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53725.log 2024-09-22T01:01:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101172.log 2024-09-22T01:01:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53725.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101063.log 2024-09-22T01:01:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101172.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128069.log 2024-09-22T01:01:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101063.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112139.log 2024-09-22T01:01:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.128069.log: 87.7% 87.5%/var/log/ceph/cluster1-client.mirror.112139.log: -- replaced with /var/log/ceph/cluster2-client.admin.40356.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.40353.log.gz 2024-09-22T01:01:53.119 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:01:53.119 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.40357.log.gz 2024-09-22T01:01:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133308.log 2024-09-22T01:01:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60977.log 2024-09-22T01:01:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133308.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42938.log 2024-09-22T01:01:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51866.log 2024-09-22T01:01:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60977.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.41290.log.gz 2024-09-22T01:01:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42938.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69674.log 2024-09-22T01:01:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51866.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38090.log 2024-09-22T01:01:53.119 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52445.log 2024-09-22T01:01:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.38090.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108354.log 2024-09-22T01:01:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.52445.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34118.log 2024-09-22T01:01:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.108354.log: /var/log/ceph/cluster1-client.mirror.34118.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99583.log 2024-09-22T01:01:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78844.log 2024-09-22T01:01:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32542.log 2024-09-22T01:01:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78844.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46088.log 2024-09-22T01:01:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89497.log 2024-09-22T01:01:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82498.log 2024-09-22T01:01:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89497.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135588.log 2024-09-22T01:01:53.120 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115039.log.gz 2024-09-22T01:01:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91674.log 2024-09-22T01:01:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135588.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133608.log 2024-09-22T01:01:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123509.log 2024-09-22T01:01:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38854.log 2024-09-22T01:01:53.120 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91570.log 2024-09-22T01:01:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38854.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125273.log 2024-09-22T01:01:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91570.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74092.log 2024-09-22T01:01:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125273.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141275.log 2024-09-22T01:01:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.74092.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42115.log 2024-09-22T01:01:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.141275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82626.log 2024-09-22T01:01:53.121 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141275.log.gz 2024-09-22T01:01:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42115.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46021.log 2024-09-22T01:01:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85926.log 2024-09-22T01:01:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63902.log 2024-09-22T01:01:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.85926.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135503.log 2024-09-22T01:01:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.63902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36048.log 2024-09-22T01:01:53.121 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47453.log.gz 2024-09-22T01:01:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34199.log 2024-09-22T01:01:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36048.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60487.log 2024-09-22T01:01:53.121 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/cluster1-client.mirror.34199.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68723.log 2024-09-22T01:01:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103062.log 2024-09-22T01:01:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59272.log 2024-09-22T01:01:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91253.log 2024-09-22T01:01:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59272.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119007.log 2024-09-22T01:01:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91253.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119465.log 2024-09-22T01:01:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.119007.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53751.log 2024-09-22T01:01:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.119465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83795.log 2024-09-22T01:01:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36111.log 2024-09-22T01:01:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83795.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.47581.log.gz 2024-09-22T01:01:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84943.log 2024-09-22T01:01:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36111.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43634.log 2024-09-22T01:01:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.84943.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111569.log 2024-09-22T01:01:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/cluster1-client.mirror.43634.log: -5 --verbose -- /var/log/ceph/cluster2-client.admin.63194.log 2024-09-22T01:01:53.122 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129577.log 2024-09-22T01:01:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.63194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111779.log 2024-09-22T01:01:53.123 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63194.log.gz 2024-09-22T01:01:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129577.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77019.log 2024-09-22T01:01:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111779.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81828.log 2024-09-22T01:01:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77019.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.67046.log.gz 2024-09-22T01:01:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121304.log 2024-09-22T01:01:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81828.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114869.log 2024-09-22T01:01:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121304.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118119.log 2024-09-22T01:01:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52154.log 2024-09-22T01:01:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114869.log: /var/log/ceph/cluster2-client.admin.118119.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38915.log 2024-09-22T01:01:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86221.log 2024-09-22T01:01:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.52154.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52154.log.gz 2024-09-22T01:01:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38915.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117453.log 2024-09-22T01:01:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129014.log 2024-09-22T01:01:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.86221.log: /var/log/ceph/cluster1-client.admin.117453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36846.log 2024-09-22T01:01:53.123 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.129014.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71203.log 2024-09-22T01:01:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36846.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111822.log 2024-09-22T01:01:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71203.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103271.log 2024-09-22T01:01:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111822.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57424.log 2024-09-22T01:01:53.124 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46571.log.gz 2024-09-22T01:01:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46826.log 2024-09-22T01:01:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103271.log: /var/log/ceph/cluster1-client.mirror.57424.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118383.log 2024-09-22T01:01:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76268.log 2024-09-22T01:01:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46826.log: /var/log/ceph/cluster2-client.admin.118383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81909.log 2024-09-22T01:01:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47496.log 2024-09-22T01:01:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76268.log: /var/log/ceph/cluster1-client.mirror.81909.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115499.log 2024-09-22T01:01:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47496.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108951.log 2024-09-22T01:01:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115499.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32581.log 2024-09-22T01:01:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97718.log 2024-09-22T01:01:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.108951.log: /var/log/ceph/cluster1-client.mirror.32581.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53513.log 2024-09-22T01:01:53.124 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.97718.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48422.log 2024-09-22T01:01:53.125 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68622.log.gz 2024-09-22T01:01:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.53513.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123276.log 2024-09-22T01:01:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48422.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110906.log 2024-09-22T01:01:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123276.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121601.log 2024-09-22T01:01:53.125 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68682.log.gz 2024-09-22T01:01:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65126.log 2024-09-22T01:01:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105373.log 2024-09-22T01:01:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65126.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82458.log 2024-09-22T01:01:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105373.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141359.log 2024-09-22T01:01:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82458.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79514.log 2024-09-22T01:01:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.141359.log: gzip -5 --verbose -- 0.0% /var/log/ceph/cluster1-client.mirror.37688.log -- replaced with /var/log/ceph/cluster2-client.admin.141359.log.gz 2024-09-22T01:01:53.125 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:01:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109018.log 2024-09-22T01:01:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37688.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130601.log 2024-09-22T01:01:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109018.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130296.log 2024-09-22T01:01:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.130601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100350.log 2024-09-22T01:01:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.130296.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136597.log 2024-09-22T01:01:53.126 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130296.log.gz 2024-09-22T01:01:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141725.log 2024-09-22T01:01:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136597.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102618.log 2024-09-22T01:01:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.141725.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47621.log 2024-09-22T01:01:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102618.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.141725.log.gz 2024-09-22T01:01:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36008.log 2024-09-22T01:01:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47621.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85765.log 2024-09-22T01:01:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36008.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69733.log 2024-09-22T01:01:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.85765.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92723.log 2024-09-22T01:01:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69733.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30279.log 2024-09-22T01:01:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94490.log 2024-09-22T01:01:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.30279.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25480.log 2024-09-22T01:01:53.126 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.94490.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67086.log 2024-09-22T01:01:53.127 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94490.log.gz 2024-09-22T01:01:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.25480.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85357.log 2024-09-22T01:01:53.127 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25480.log.gz 2024-09-22T01:01:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67086.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104846.log 2024-09-22T01:01:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.85357.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85357.log.gz 2024-09-22T01:01:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77244.log 2024-09-22T01:01:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104846.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63146.log 2024-09-22T01:01:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77244.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127437.log 2024-09-22T01:01:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.63146.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67572.log 2024-09-22T01:01:53.127 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63146.log.gz 2024-09-22T01:01:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84026.log 2024-09-22T01:01:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67572.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93689.log 2024-09-22T01:01:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.84026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127226.log 2024-09-22T01:01:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93689.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59481.log 2024-09-22T01:01:53.127 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86619.log 2024-09-22T01:01:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59481.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62749.log 2024-09-22T01:01:53.128 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69143.log.gz 2024-09-22T01:01:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.86619.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92512.log 2024-09-22T01:01:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62749.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127377.log 2024-09-22T01:01:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39551.log 2024-09-22T01:01:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.127377.log: --verbose -- /var/log/ceph/cluster1-client.admin.116965.log 2024-09-22T01:01:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39551.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108012.log 2024-09-22T01:01:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.116965.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.116965.log.gz 2024-09-22T01:01:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33948.log 2024-09-22T01:01:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.108012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65639.log 2024-09-22T01:01:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33948.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54553.log 2024-09-22T01:01:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65639.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93060.log 2024-09-22T01:01:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54553.log: /var/log/ceph/cluster1-client.mirror.93060.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69314.log.gz 2024-09-22T01:01:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45012.log 2024-09-22T01:01:53.128 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64987.log 2024-09-22T01:01:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94503.log 2024-09-22T01:01:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64987.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135085.log 2024-09-22T01:01:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67782.log 2024-09-22T01:01:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121778.log 2024-09-22T01:01:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67782.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142234.log 2024-09-22T01:01:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121778.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122544.log 2024-09-22T01:01:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.142234.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119974.log 2024-09-22T01:01:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122544.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.142234.log.gz 2024-09-22T01:01:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125754.log 2024-09-22T01:01:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.119974.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56078.log 2024-09-22T01:01:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125754.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66876.log 2024-09-22T01:01:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56078.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53624.log 2024-09-22T01:01:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66876.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69882.log 2024-09-22T01:01:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53624.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44637.log 2024-09-22T01:01:53.129 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83244.log 2024-09-22T01:01:53.130 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90269.log.gz 2024-09-22T01:01:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134288.log 2024-09-22T01:01:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.83244.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137442.log 2024-09-22T01:01:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134288.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129465.log 2024-09-22T01:01:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69253.log 2024-09-22T01:01:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.129465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81051.log 2024-09-22T01:01:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69253.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137692.log 2024-09-22T01:01:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81051.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134370.log 2024-09-22T01:01:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103753.log 2024-09-22T01:01:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134370.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81741.log 2024-09-22T01:01:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77535.log 2024-09-22T01:01:53.130 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42031.log 2024-09-22T01:01:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77535.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95437.log 2024-09-22T01:01:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.42031.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40141.log 2024-09-22T01:01:53.131 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.38788.log.gz 2024-09-22T01:01:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.95437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105372.log 2024-09-22T01:01:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40141.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31201.log 2024-09-22T01:01:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105372.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90159.log 2024-09-22T01:01:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63788.log 2024-09-22T01:01:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31201.log: /var/log/ceph/cluster1-client.mirror.90159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33569.log 2024-09-22T01:01:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.63788.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128830.log 2024-09-22T01:01:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33569.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103482.log 2024-09-22T01:01:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128830.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139386.log 2024-09-22T01:01:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103482.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85400.log 2024-09-22T01:01:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139386.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83107.log 2024-09-22T01:01:53.131 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.139386.log.gz 2024-09-22T01:01:53.131 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.85400.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85400.log.gz 2024-09-22T01:01:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82329.log 2024-09-22T01:01:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.83107.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65158.log 2024-09-22T01:01:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82329.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107728.log 2024-09-22T01:01:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65158.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53816.log 2024-09-22T01:01:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.107728.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99144.log 2024-09-22T01:01:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53816.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84734.log 2024-09-22T01:01:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99144.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115247.log 2024-09-22T01:01:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.84734.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105664.log 2024-09-22T01:01:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48965.log 2024-09-22T01:01:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115247.log: /var/log/ceph/cluster1-client.mirror.105664.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114406.log 2024-09-22T01:01:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48965.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53102.log 2024-09-22T01:01:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114406.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48421.log 2024-09-22T01:01:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.53102.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93353.log 2024-09-22T01:01:53.132 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89497.log.gz 2024-09-22T01:01:53.132 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48421.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108334.log 2024-09-22T01:01:53.133 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.89488.log.gz 2024-09-22T01:01:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47663.log 2024-09-22T01:01:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.108334.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76944.log 2024-09-22T01:01:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47663.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84659.log 2024-09-22T01:01:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68303.log 2024-09-22T01:01:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110589.log 2024-09-22T01:01:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68303.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85039.log 2024-09-22T01:01:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110589.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79135.log 2024-09-22T01:01:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.85039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111396.log 2024-09-22T01:01:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79135.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71122.log 2024-09-22T01:01:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88084.log 2024-09-22T01:01:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45306.log 2024-09-22T01:01:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55491.log 2024-09-22T01:01:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49949.log 2024-09-22T01:01:53.133 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55491.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92639.log 2024-09-22T01:01:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137862.log 2024-09-22T01:01:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92639.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110562.log 2024-09-22T01:01:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137862.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118942.log 2024-09-22T01:01:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39073.log 2024-09-22T01:01:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.118942.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86241.log 2024-09-22T01:01:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54171.log 2024-09-22T01:01:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.86241.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127880.log 2024-09-22T01:01:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54171.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131212.log 2024-09-22T01:01:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58684.log 2024-09-22T01:01:53.134 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64652.log.gz 2024-09-22T01:01:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131212.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72893.log 2024-09-22T01:01:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58684.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54874.log 2024-09-22T01:01:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72893.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114594.log 2024-09-22T01:01:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54874.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44847.log 2024-09-22T01:01:53.134 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58807.log 2024-09-22T01:01:53.135 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114594.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98064.log 2024-09-22T01:01:53.135 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44847.log: /var/log/ceph/cluster1-client.mirror.58807.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41913.log 2024-09-22T01:01:53.135 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.98064.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108837.log 2024-09-22T01:01:53.135 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41913.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50207.log 2024-09-22T01:01:53.135 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.108837.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54294.log 2024-09-22T01:01:53.135 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50207.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56833.log 2024-09-22T01:01:53.135 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54294.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41819.log 2024-09-22T01:01:53.135 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73582.log 2024-09-22T01:01:53.135 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125336.log 2024-09-22T01:01:53.135 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.73582.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29491.log 2024-09-22T01:01:53.135 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54066.log 2024-09-22T01:01:53.135 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130198.log 2024-09-22T01:01:53.136 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29491.log.gz 2024-09-22T01:01:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.54066.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104718.log 2024-09-22T01:01:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.130198.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134209.log 2024-09-22T01:01:53.136 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130198.log.gz 2024-09-22T01:01:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104718.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33088.log 2024-09-22T01:01:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134209.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64586.log 2024-09-22T01:01:53.136 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67255.log.gz 2024-09-22T01:01:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115921.log 2024-09-22T01:01:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64586.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133699.log 2024-09-22T01:01:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115921.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60109.log 2024-09-22T01:01:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133699.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140274.log 2024-09-22T01:01:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60109.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108240.log 2024-09-22T01:01:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140274.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132640.log 2024-09-22T01:01:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.108240.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102408.log 2024-09-22T01:01:53.136 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.132640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117158.log 2024-09-22T01:01:53.137 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102408.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100731.log 2024-09-22T01:01:53.137 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.117158.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140505.log 2024-09-22T01:01:53.137 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100731.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68832.log 2024-09-22T01:01:53.137 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140505.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103060.log.gz 2024-09-22T01:01:53.334 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124987.log 2024-09-22T01:01:53.334 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103062.log.gz 2024-09-22T01:01:53.334 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94915.log 2024-09-22T01:01:53.334 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124987.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126112.log 2024-09-22T01:01:53.334 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.94915.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120108.log 2024-09-22T01:01:53.334 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133478.log 2024-09-22T01:01:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.120108.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65794.log 2024-09-22T01:01:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133478.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130917.log 2024-09-22T01:01:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65794.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110079.log 2024-09-22T01:01:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.130917.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136343.log 2024-09-22T01:01:53.335 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.92429.log.gz 2024-09-22T01:01:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110079.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46357.log 2024-09-22T01:01:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136343.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124853.log 2024-09-22T01:01:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46357.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133567.log 2024-09-22T01:01:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124853.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86335.log 2024-09-22T01:01:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133567.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35168.log 2024-09-22T01:01:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.86335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115352.log 2024-09-22T01:01:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123819.log 2024-09-22T01:01:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56541.log 2024-09-22T01:01:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115352.log: /var/log/ceph/cluster1-client.mirror.123819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123803.log 2024-09-22T01:01:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54788.log 2024-09-22T01:01:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123803.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101801.log 2024-09-22T01:01:53.336 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91736.log.gz 2024-09-22T01:01:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77959.log 2024-09-22T01:01:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54788.log: /var/log/ceph/cluster1-client.mirror.101801.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49685.log 2024-09-22T01:01:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77959.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79389.log 2024-09-22T01:01:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49685.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67424.log 2024-09-22T01:01:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123513.log 2024-09-22T01:01:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67424.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104216.log 2024-09-22T01:01:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123513.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107631.log 2024-09-22T01:01:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104216.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-09-22T01:01:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.107631.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136428.log 2024-09-22T01:01:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43391.log 2024-09-22T01:01:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136428.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97665.log 2024-09-22T01:01:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43391.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55443.log 2024-09-22T01:01:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103794.log 2024-09-22T01:01:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.97665.log: /var/log/ceph/cluster1-client.mirror.55443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110265.log 2024-09-22T01:01:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98891.log 2024-09-22T01:01:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103794.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122480.log.gz 2024-09-22T01:01:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110265.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69943.log 2024-09-22T01:01:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.98891.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105056.log 2024-09-22T01:01:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69943.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131563.log 2024-09-22T01:01:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125903.log 2024-09-22T01:01:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131563.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119201.log 2024-09-22T01:01:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125903.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136469.log 2024-09-22T01:01:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.119201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120290.log 2024-09-22T01:01:53.337 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119201.log.gz 2024-09-22T01:01:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136469.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.105222.log.gz 2024-09-22T01:01:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109784.log 2024-09-22T01:01:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.120290.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54469.log 2024-09-22T01:01:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109784.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133700.log 2024-09-22T01:01:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54469.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76079.log 2024-09-22T01:01:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132756.log 2024-09-22T01:01:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140911.log 2024-09-22T01:01:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.76079.log: /var/log/ceph/cluster1-client.mirror.132756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52740.log 2024-09-22T01:01:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140911.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99397.log 2024-09-22T01:01:53.338 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121302.log.gz 2024-09-22T01:01:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35501.log 2024-09-22T01:01:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99397.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62750.log 2024-09-22T01:01:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35501.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130642.log 2024-09-22T01:01:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62750.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90059.log 2024-09-22T01:01:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.130642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131118.log 2024-09-22T01:01:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90059.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137397.log 2024-09-22T01:01:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.131118.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112559.log 2024-09-22T01:01:53.339 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123090.log.gz 2024-09-22T01:01:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137397.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74243.log 2024-09-22T01:01:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120202.log 2024-09-22T01:01:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.74243.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37182.log 2024-09-22T01:01:53.339 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74243.log.gz 2024-09-22T01:01:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.120202.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106926.log 2024-09-22T01:01:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70932.log 2024-09-22T01:01:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.106926.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107891.log 2024-09-22T01:01:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70932.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128530.log 2024-09-22T01:01:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107891.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41571.log 2024-09-22T01:01:53.339 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.107891.log.gz 2024-09-22T01:01:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128530.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82330.log 2024-09-22T01:01:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41571.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62646.log 2024-09-22T01:01:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82330.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104929.log 2024-09-22T01:01:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62646.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73161.log 2024-09-22T01:01:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104929.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43777.log 2024-09-22T01:01:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105717.log 2024-09-22T01:01:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73161.log: /var/log/ceph/cluster1-client.mirror.43777.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113124.log 2024-09-22T01:01:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45223.log 2024-09-22T01:01:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.105717.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124497.log.gz 2024-09-22T01:01:53.340 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105717.log.gz 2024-09-22T01:01:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113124.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117580.log 2024-09-22T01:01:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45223.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103332.log 2024-09-22T01:01:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.117580.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58766.log 2024-09-22T01:01:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116340.log 2024-09-22T01:01:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94342.log 2024-09-22T01:01:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86154.log 2024-09-22T01:01:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.94342.log: gzip 0.0% -5 -- replaced with /var/log/ceph/cluster1-client.admin.94342.log.gz 2024-09-22T01:01:53.340 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.74278.log 2024-09-22T01:01:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.86154.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126913.log.gz 2024-09-22T01:01:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47622.log 2024-09-22T01:01:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.74278.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74278.log.gz 2024-09-22T01:01:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61096.log 2024-09-22T01:01:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47622.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108582.log 2024-09-22T01:01:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37393.log 2024-09-22T01:01:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.108582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44216.log 2024-09-22T01:01:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37393.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39134.log 2024-09-22T01:01:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53689.log 2024-09-22T01:01:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39134.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36218.log 2024-09-22T01:01:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53689.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45729.log 2024-09-22T01:01:53.341 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127880.log.gz 2024-09-22T01:01:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93604.log 2024-09-22T01:01:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45729.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86355.log 2024-09-22T01:01:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104742.log 2024-09-22T01:01:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5/var/log/ceph/cluster1-client.admin.86355.log: --verbose -- /var/log/ceph/cluster1-client.mirror.48588.log 2024-09-22T01:01:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69984.log 2024-09-22T01:01:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48588.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57296.log 2024-09-22T01:01:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69984.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37059.log 2024-09-22T01:01:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57296.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122672.log 2024-09-22T01:01:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37059.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140916.log 2024-09-22T01:01:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122672.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48755.log 2024-09-22T01:01:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65369.log 2024-09-22T01:01:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140916.log: /var/log/ceph/cluster1-client.mirror.48755.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96770.log 2024-09-22T01:01:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97437.log 2024-09-22T01:01:53.342 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.87594.log.gz 2024-09-22T01:01:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96770.log: 0.0%gzip -- replaced with /var/log/ceph/cluster2-client.admin.96770.log.gz 2024-09-22T01:01:53.342 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40927.log 2024-09-22T01:01:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.97437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102642.log 2024-09-22T01:01:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.40927.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130575.log 2024-09-22T01:01:53.342 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.87695.log.gz 2024-09-22T01:01:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93860.log 2024-09-22T01:01:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.130575.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38634.log 2024-09-22T01:01:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93860.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66877.log 2024-09-22T01:01:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38634.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91800.log 2024-09-22T01:01:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91168.log 2024-09-22T01:01:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41042.log 2024-09-22T01:01:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92680.log 2024-09-22T01:01:53.343 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99583.log.gz 2024-09-22T01:01:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41042.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64357.log 2024-09-22T01:01:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92680.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83191.log 2024-09-22T01:01:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64357.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122962.log 2024-09-22T01:01:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107257.log 2024-09-22T01:01:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83191.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.83191.log.gz 2024-09-22T01:01:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122962.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91883.log 2024-09-22T01:01:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107257.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112622.log 2024-09-22T01:01:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121724.log 2024-09-22T01:01:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112622.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84785.log 2024-09-22T01:01:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121724.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130236.log 2024-09-22T01:01:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109065.log 2024-09-22T01:01:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.130236.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67128.log 2024-09-22T01:01:53.344 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130236.log.gz 2024-09-22T01:01:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.109065.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95897.log 2024-09-22T01:01:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70027.log 2024-09-22T01:01:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.95897.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92368.log 2024-09-22T01:01:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.70027.log: --verbose -- /var/log/ceph/cluster1-client.admin.119237.log 2024-09-22T01:01:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92368.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36403.log 2024-09-22T01:01:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.119237.log: gzip 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138029.log.gz 2024-09-22T01:01:53.344 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.29880.log 2024-09-22T01:01:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36403.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60978.log 2024-09-22T01:01:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.29880.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29880.log.gz 2024-09-22T01:01:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28770.log 2024-09-22T01:01:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67214.log 2024-09-22T01:01:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.28770.log: gzip 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28770.log.gz 2024-09-22T01:01:53.345 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93396.log 2024-09-22T01:01:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67214.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116341.log 2024-09-22T01:01:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49135.log 2024-09-22T01:01:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142436.log 2024-09-22T01:01:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49135.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135653.log 2024-09-22T01:01:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.142436.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136220.log 2024-09-22T01:01:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135653.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.142436.log.gz 2024-09-22T01:01:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55182.log 2024-09-22T01:01:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136220.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119008.log 2024-09-22T01:01:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68097.log 2024-09-22T01:01:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.119008.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38087.log 2024-09-22T01:01:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68097.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60446.log 2024-09-22T01:01:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.38087.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140332.log 2024-09-22T01:01:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60446.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.93689.log.gz 2024-09-22T01:01:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125796.log 2024-09-22T01:01:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50710.log 2024-09-22T01:01:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125796.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125483.log 2024-09-22T01:01:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89850.log 2024-09-22T01:01:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125483.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107893.log 2024-09-22T01:01:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119693.log 2024-09-22T01:01:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107893.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.107893.log.gz 2024-09-22T01:01:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86013.log 2024-09-22T01:01:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.119693.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88465.log 2024-09-22T01:01:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.86013.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127291.log 2024-09-22T01:01:53.346 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.83620.log.gz 2024-09-22T01:01:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88851.log 2024-09-22T01:01:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127291.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34661.log 2024-09-22T01:01:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.88851.log: -- /var/log/ceph/cluster1-client.mirror.99750.log 2024-09-22T01:01:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34661.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81406.log 2024-09-22T01:01:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99750.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65343.log 2024-09-22T01:01:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81406.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90372.log 2024-09-22T01:01:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65343.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79388.log 2024-09-22T01:01:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90372.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92997.log 2024-09-22T01:01:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79388.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104382.log 2024-09-22T01:01:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92997.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66794.log 2024-09-22T01:01:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61045.log 2024-09-22T01:01:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66794.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122307.log 2024-09-22T01:01:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42333.log 2024-09-22T01:01:53.347 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.61045.log.gz 2024-09-22T01:01:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101947.log 2024-09-22T01:01:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134917.log 2024-09-22T01:01:53.348 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84906.log.gz 2024-09-22T01:01:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101947.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134622.log 2024-09-22T01:01:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134917.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36509.log 2024-09-22T01:01:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134622.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.85042.log.gz 2024-09-22T01:01:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129744.log 2024-09-22T01:01:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90270.log 2024-09-22T01:01:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129744.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43048.log 2024-09-22T01:01:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30335.log 2024-09-22T01:01:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43048.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60826.log 2024-09-22T01:01:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38698.log 2024-09-22T01:01:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60826.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114763.log 2024-09-22T01:01:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53433.log 2024-09-22T01:01:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114763.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70993.log 2024-09-22T01:01:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53433.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101527.log 2024-09-22T01:01:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70993.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-09-22T01:01:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101527.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133607.log 2024-09-22T01:01:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54404.log 2024-09-22T01:01:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133607.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41973.log 2024-09-22T01:01:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54404.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101778.log 2024-09-22T01:01:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41973.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63861.log 2024-09-22T01:01:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101778.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142817.log 2024-09-22T01:01:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.63861.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34328.log 2024-09-22T01:01:53.349 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48755.log.gz 2024-09-22T01:01:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.142817.log: gzip 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142817.log.gz -5 2024-09-22T01:01:53.349 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.137315.log 2024-09-22T01:01:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60403.log 2024-09-22T01:01:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137315.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48043.log 2024-09-22T01:01:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60403.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138281.log 2024-09-22T01:01:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78255.log 2024-09-22T01:01:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138281.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45559.log 2024-09-22T01:01:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78255.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115434.log 2024-09-22T01:01:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45559.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90999.log 2024-09-22T01:01:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115434.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31814.log 2024-09-22T01:01:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109932.log 2024-09-22T01:01:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90999.log: /var/log/ceph/cluster1-client.mirror.31814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98407.log 2024-09-22T01:01:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.109932.log: --verbose -- /var/log/ceph/cluster1-client.mirror.45096.log 2024-09-22T01:01:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98407.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49052.log 2024-09-22T01:01:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45096.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74631.log 2024-09-22T01:01:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49052.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135337.log 2024-09-22T01:01:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.74631.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118435.log 2024-09-22T01:01:53.350 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128069.log.gz 2024-09-22T01:01:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135337.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142181.log 2024-09-22T01:01:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.118435.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127998.log 2024-09-22T01:01:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.142181.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125902.log 2024-09-22T01:01:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.127998.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87317.log 2024-09-22T01:01:53.351 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.127998.log.gz 2024-09-22T01:01:53.351 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.142181.log.gz 2024-09-22T01:01:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45642.log 2024-09-22T01:01:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.87317.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66942.log 2024-09-22T01:01:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50328.log 2024-09-22T01:01:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66942.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95866.log 2024-09-22T01:01:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69884.log 2024-09-22T01:01:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124770.log 2024-09-22T01:01:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.95866.log: /var/log/ceph/cluster1-client.mirror.69884.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118868.log 2024-09-22T01:01:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68766.log 2024-09-22T01:01:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118868.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40008.log 2024-09-22T01:01:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80105.log 2024-09-22T01:01:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40008.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136979.log 2024-09-22T01:01:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39858.log 2024-09-22T01:01:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136979.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112245.log 2024-09-22T01:01:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67087.log 2024-09-22T01:01:53.352 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.132471.log.gz 2024-09-22T01:01:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112245.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129579.log 2024-09-22T01:01:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39858.log: /var/log/ceph/cluster1-client.mirror.67087.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65837.log 2024-09-22T01:01:53.352 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.132640.log.gz 2024-09-22T01:01:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133992.log 2024-09-22T01:01:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65837.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131944.log 2024-09-22T01:01:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63560.log 2024-09-22T01:01:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59062.log 2024-09-22T01:01:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.63560.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103586.log 2024-09-22T01:01:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113819.log 2024-09-22T01:01:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103586.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102124.log 2024-09-22T01:01:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93209.log 2024-09-22T01:01:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102124.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69252.log 2024-09-22T01:01:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93209.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34117.log 2024-09-22T01:01:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69252.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57002.log 2024-09-22T01:01:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34117.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112516.log 2024-09-22T01:01:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115183.log 2024-09-22T01:01:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112516.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90034.log 2024-09-22T01:01:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60361.log 2024-09-22T01:01:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90034.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108468.log 2024-09-22T01:01:53.353 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57842.log.gz 2024-09-22T01:01:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60361.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137944.log 2024-09-22T01:01:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.108468.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136302.log 2024-09-22T01:01:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97987.log 2024-09-22T01:01:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136302.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71562.log 2024-09-22T01:01:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.97987.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122105.log 2024-09-22T01:01:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104426.log 2024-09-22T01:01:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122105.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37391.log 2024-09-22T01:01:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126499.log 2024-09-22T01:01:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37391.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54830.log 2024-09-22T01:01:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126499.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52719.log 2024-09-22T01:01:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54830.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64626.log 2024-09-22T01:01:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52719.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36551.log 2024-09-22T01:01:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64626.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52491.log 2024-09-22T01:01:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36551.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49519.log 2024-09-22T01:01:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62429.log 2024-09-22T01:01:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75651.log 2024-09-22T01:01:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.62429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93478.log 2024-09-22T01:01:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81576.log 2024-09-22T01:01:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93478.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104637.log 2024-09-22T01:01:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81576.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.133700.log.gz 2024-09-22T01:01:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118260.log 2024-09-22T01:01:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139296.log 2024-09-22T01:01:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.118260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114198.log 2024-09-22T01:01:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.139296.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45097.log 2024-09-22T01:01:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114198.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99229.log 2024-09-22T01:01:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45097.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.86013.log.gz 2024-09-22T01:01:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103480.log 2024-09-22T01:01:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99229.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61987.log 2024-09-22T01:01:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103480.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67464.log 2024-09-22T01:01:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.61987.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60953.log 2024-09-22T01:01:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58432.log 2024-09-22T01:01:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.60953.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33989.log 2024-09-22T01:01:53.355 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.60953.log.gz 2024-09-22T01:01:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58432.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53196.log 2024-09-22T01:01:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63996.log 2024-09-22T01:01:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53196.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90875.log 2024-09-22T01:01:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63996.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34092.log 2024-09-22T01:01:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72599.log 2024-09-22T01:01:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34092.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96168.log 2024-09-22T01:01:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72599.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54108.log 2024-09-22T01:01:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.96168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106262.log 2024-09-22T01:01:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54108.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82119.log 2024-09-22T01:01:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106262.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57547.log 2024-09-22T01:01:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82119.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81785.log 2024-09-22T01:01:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57547.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36088.log 2024-09-22T01:01:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122921.log 2024-09-22T01:01:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123026.log 2024-09-22T01:01:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122921.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86940.log 2024-09-22T01:01:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51182.log 2024-09-22T01:01:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123026.log: /var/log/ceph/cluster1-client.mirror.86940.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95275.log 2024-09-22T01:01:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.51182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80146.log 2024-09-22T01:01:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.95275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33528.log 2024-09-22T01:01:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62491.log 2024-09-22T01:01:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33528.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127500.log 2024-09-22T01:01:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.62491.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115499.log.gz 2024-09-22T01:01:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69102.log 2024-09-22T01:01:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127500.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94812.log 2024-09-22T01:01:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69102.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60322.log 2024-09-22T01:01:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.94812.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135758.log 2024-09-22T01:01:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106587.log 2024-09-22T01:01:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135758.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56961.log 2024-09-22T01:01:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106587.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57883.log 2024-09-22T01:01:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56961.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105223.log 2024-09-22T01:01:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138114.log 2024-09-22T01:01:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105223.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108609.log 2024-09-22T01:01:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138114.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98787.log 2024-09-22T01:01:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.108609.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137271.log 2024-09-22T01:01:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72289.log 2024-09-22T01:01:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101697.log 2024-09-22T01:01:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.72289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47749.log 2024-09-22T01:01:53.358 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53492.log.gz 2024-09-22T01:01:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101697.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.53513.log.gz 2024-09-22T01:01:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71538.log 2024-09-22T01:01:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47749.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71123.log 2024-09-22T01:01:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71538.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128598.log 2024-09-22T01:01:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71123.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66834.log 2024-09-22T01:01:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128598.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127752.log 2024-09-22T01:01:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66834.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141313.log 2024-09-22T01:01:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57633.log 2024-09-22T01:01:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.141313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57800.log 2024-09-22T01:01:53.359 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141313.log.gz 2024-09-22T01:01:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57633.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54110.log 2024-09-22T01:01:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58474.log 2024-09-22T01:01:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115644.log 2024-09-22T01:01:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58474.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131817.log 2024-09-22T01:01:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38269.log 2024-09-22T01:01:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131817.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51751.log 2024-09-22T01:01:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.38269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68412.log 2024-09-22T01:01:53.359 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.38634.log.gz 2024-09-22T01:01:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108106.log 2024-09-22T01:01:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68412.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97376.log 2024-09-22T01:01:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.108106.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48881.log 2024-09-22T01:01:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.97376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113191.log 2024-09-22T01:01:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46164.log 2024-09-22T01:01:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113191.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134916.log 2024-09-22T01:01:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46164.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35502.log 2024-09-22T01:01:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134916.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66120.log 2024-09-22T01:01:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35502.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85428.log 2024-09-22T01:01:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66120.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111283.log 2024-09-22T01:01:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.85428.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85428.log.gz 2024-09-22T01:01:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86200.log 2024-09-22T01:01:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94813.log 2024-09-22T01:01:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.86200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47882.log 2024-09-22T01:01:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.94813.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51392.log 2024-09-22T01:01:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57086.log 2024-09-22T01:01:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51392.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134831.log 2024-09-22T01:01:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57086.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128034.log 2024-09-22T01:01:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54214.log 2024-09-22T01:01:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128034.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42394.log 2024-09-22T01:01:53.361 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128034.log.gz 2024-09-22T01:01:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.54214.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57338.log 2024-09-22T01:01:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42394.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115457.log 2024-09-22T01:01:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57338.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137443.log 2024-09-22T01:01:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115457.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119787.log 2024-09-22T01:01:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137443.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55115.log.gz 2024-09-22T01:01:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57044.log 2024-09-22T01:01:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.119787.log: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.74786.log.gz 2024-09-22T01:01:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117451.log 2024-09-22T01:01:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57044.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40177.log 2024-09-22T01:01:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.117451.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44929.log 2024-09-22T01:01:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86745.log 2024-09-22T01:01:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44929.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110460.log 2024-09-22T01:01:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.86745.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139130.log 2024-09-22T01:01:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141410.log 2024-09-22T01:01:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139130.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.139130.log.gz 2024-09-22T01:01:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116820.log 2024-09-22T01:01:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.141410.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83205.log 2024-09-22T01:01:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110016.log 2024-09-22T01:01:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83205.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103900.log 2024-09-22T01:01:53.362 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.83205.log.gz 2024-09-22T01:01:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110016.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108085.log 2024-09-22T01:01:53.362 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.72226.log.gz 2024-09-22T01:01:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103900.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142869.log 2024-09-22T01:01:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.108085.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39802.log 2024-09-22T01:01:53.363 INFO:teuthology.orchestra.run.smithi060.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.141410.log.gz 2024-09-22T01:01:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.142869.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50761.log 2024-09-22T01:01:53.363 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142869.log.gz 2024-09-22T01:01:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120129.log 2024-09-22T01:01:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50761.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72340.log.gz 2024-09-22T01:01:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79347.log 2024-09-22T01:01:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120129.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130064.log 2024-09-22T01:01:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79347.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108769.log 2024-09-22T01:01:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.130064.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29955.log 2024-09-22T01:01:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.108769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94066.log 2024-09-22T01:01:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.29955.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139262.log 2024-09-22T01:01:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94066.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49371.log 2024-09-22T01:01:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.139262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86294.log 2024-09-22T01:01:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49371.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.118017.log.gz 2024-09-22T01:01:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74302.log 2024-09-22T01:01:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.86294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105013.log 2024-09-22T01:01:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.74302.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74302.log.gz 2024-09-22T01:01:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129114.log 2024-09-22T01:01:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105013.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46742.log 2024-09-22T01:01:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129114.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110352.log 2024-09-22T01:01:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44341.log 2024-09-22T01:01:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.110352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118319.log 2024-09-22T01:01:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100856.log 2024-09-22T01:01:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.118319.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28951.log 2024-09-22T01:01:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100856.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79809.log 2024-09-22T01:01:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.28951.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28951.log.gz 2024-09-22T01:01:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109889.log 2024-09-22T01:01:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79809.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55911.log 2024-09-22T01:01:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.109889.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68767.log 2024-09-22T01:01:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55911.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59227.log 2024-09-22T01:01:53.365 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.97437.log.gz 2024-09-22T01:01:53.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68767.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123657.log 2024-09-22T01:01:53.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59227.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70613.log 2024-09-22T01:01:53.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123657.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58262.log 2024-09-22T01:01:53.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70613.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112895.log 2024-09-22T01:01:53.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58262.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66835.log 2024-09-22T01:01:53.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115079.log 2024-09-22T01:01:53.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66835.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119834.log 2024-09-22T01:01:53.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115079.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120413.log 2024-09-22T01:01:53.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.119834.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107699.log 2024-09-22T01:01:53.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100227.log 2024-09-22T01:01:53.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.107699.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106114.log 2024-09-22T01:01:53.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100227.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59271.log 2024-09-22T01:01:53.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106114.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80565.log 2024-09-22T01:01:53.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71124.log 2024-09-22T01:01:53.366 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56791.log.gz 2024-09-22T01:01:53.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80565.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133867.log 2024-09-22T01:01:53.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71124.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58598.log 2024-09-22T01:01:53.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133867.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101652.log 2024-09-22T01:01:53.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58598.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73744.log 2024-09-22T01:01:53.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100518.log 2024-09-22T01:01:53.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73744.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70314.log 2024-09-22T01:01:53.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35080.log 2024-09-22T01:01:53.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30731.log 2024-09-22T01:01:53.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35080.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52787.log 2024-09-22T01:01:53.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30731.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71855.log 2024-09-22T01:01:53.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.52787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111825.log 2024-09-22T01:01:53.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71855.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67633.log 2024-09-22T01:01:53.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111825.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114826.log 2024-09-22T01:01:53.366 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.107254.log.gz 2024-09-22T01:01:53.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67633.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124352.log 2024-09-22T01:01:53.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114826.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47981.log 2024-09-22T01:01:53.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124352.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37183.log 2024-09-22T01:01:53.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47981.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99817.log 2024-09-22T01:01:53.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48167.log 2024-09-22T01:01:53.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99817.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95201.log 2024-09-22T01:01:53.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48167.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121919.log 2024-09-22T01:01:53.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.95201.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129258.log 2024-09-22T01:01:53.367 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.119351.log.gz 2024-09-22T01:01:53.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72598.log 2024-09-22T01:01:53.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98301.log 2024-09-22T01:01:53.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72598.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103543.log 2024-09-22T01:01:53.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98301.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128896.log 2024-09-22T01:01:53.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103543.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50375.log 2024-09-22T01:01:53.368 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.128597.log.gz 2024-09-22T01:01:53.368 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128896.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39135.log 2024-09-22T01:01:53.368 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52331.log 2024-09-22T01:01:53.368 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39135.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132276.log 2024-09-22T01:01:53.368 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.129111.log.gz 2024-09-22T01:01:53.368 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.52331.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107098.log 2024-09-22T01:01:53.368 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132276.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64880.log 2024-09-22T01:01:53.368 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.107098.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34030.log 2024-09-22T01:01:53.368 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45057.log 2024-09-22T01:01:53.368 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93628.log 2024-09-22T01:01:53.368 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92933.log 2024-09-22T01:01:53.368 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93628.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118623.log 2024-09-22T01:01:53.368 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92933.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93480.log 2024-09-22T01:01:53.368 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.118623.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53366.log 2024-09-22T01:01:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93480.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112897.log 2024-09-22T01:01:53.369 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.61325.log.gz 2024-09-22T01:01:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133908.log 2024-09-22T01:01:53.369 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.61486.log.gz 2024-09-22T01:01:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112897.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61707.log 2024-09-22T01:01:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32539.log 2024-09-22T01:01:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61707.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110653.log 2024-09-22T01:01:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32539.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111887.log 2024-09-22T01:01:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86127.log 2024-09-22T01:01:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48294.log 2024-09-22T01:01:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.86127.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63540.log 2024-09-22T01:01:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49947.log 2024-09-22T01:01:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98110.log 2024-09-22T01:01:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.49947.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.49947.log.gz 2024-09-22T01:01:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82287.log 2024-09-22T01:01:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70278.log 2024-09-22T01:01:53.370 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82287.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55576.log 2024-09-22T01:01:53.370 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70278.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130062.log 2024-09-22T01:01:53.370 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55576.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41000.log 2024-09-22T01:01:53.370 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.130062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104677.log 2024-09-22T01:01:53.370 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41000.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44636.log 2024-09-22T01:01:53.370 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104677.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45517.log 2024-09-22T01:01:53.370 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92554.log 2024-09-22T01:01:53.370 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45517.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133532.log 2024-09-22T01:01:53.370 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142973.log 2024-09-22T01:01:53.370 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92158.log 2024-09-22T01:01:53.370 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.142973.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142973.log.gz 2024-09-22T01:01:53.370 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102494.log 2024-09-22T01:01:53.370 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92158.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86314.log 2024-09-22T01:01:53.370 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102494.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42157.log 2024-09-22T01:01:53.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.86314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97417.log 2024-09-22T01:01:53.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.42157.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62049.log 2024-09-22T01:01:53.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.97417.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36259.log 2024-09-22T01:01:53.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62049.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84958.log 2024-09-22T01:01:53.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32623.log 2024-09-22T01:01:53.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.84958.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35982.log 2024-09-22T01:01:53.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32623.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87841.log 2024-09-22T01:01:53.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35982.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90875.log.gz 2024-09-22T01:01:53.371 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117837.log 2024-09-22T01:01:53.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87841.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90790.log 2024-09-22T01:01:53.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.117837.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60026.log 2024-09-22T01:01:53.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90790.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55492.log 2024-09-22T01:01:53.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138449.log 2024-09-22T01:01:53.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55492.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84332.log 2024-09-22T01:01:53.372 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138449.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101107.log 2024-09-22T01:01:53.372 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.84332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45810.log 2024-09-22T01:01:53.372 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101107.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30993.log 2024-09-22T01:01:53.372 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45810.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142389.log 2024-09-22T01:01:53.372 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46021.log.gz 2024-09-22T01:01:53.372 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100100.log 2024-09-22T01:01:53.372 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.142389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101988.log 2024-09-22T01:01:53.372 INFO:teuthology.orchestra.run.smithi060.stderr: 67.5% -- replaced with /var/log/ceph/cluster1-client.mirror.142389.log.gz 2024-09-22T01:01:53.372 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100100.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93519.log 2024-09-22T01:01:53.372 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50442.log 2024-09-22T01:01:53.372 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131459.log 2024-09-22T01:01:53.372 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88717.log 2024-09-22T01:01:53.372 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131459.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48176.log 2024-09-22T01:01:53.372 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45476.log 2024-09-22T01:01:53.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48176.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85038.log 2024-09-22T01:01:53.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32707.log 2024-09-22T01:01:53.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.85038.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70027.log.gz 2024-09-22T01:01:53.373 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110458.log 2024-09-22T01:01:53.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32707.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128571.log 2024-09-22T01:01:53.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110458.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120607.log 2024-09-22T01:01:53.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128571.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81548.log 2024-09-22T01:01:53.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120607.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104256.log 2024-09-22T01:01:53.373 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62429.log.gz 2024-09-22T01:01:53.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29989.log 2024-09-22T01:01:53.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117455.log 2024-09-22T01:01:53.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.29989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32227.log 2024-09-22T01:01:53.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.117455.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105815.log 2024-09-22T01:01:53.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32227.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94406.log 2024-09-22T01:01:53.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.105815.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62698.log.gz 2024-09-22T01:01:53.374 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67362.log 2024-09-22T01:01:53.374 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105815.log.gz 2024-09-22T01:01:53.374 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.62750.log.gz 2024-09-22T01:01:53.374 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.94406.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94406.log.gz 2024-09-22T01:01:53.374 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100896.log 2024-09-22T01:01:53.374 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85812.log 2024-09-22T01:01:53.374 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100896.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78043.log 2024-09-22T01:01:53.374 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.85812.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40809.log 2024-09-22T01:01:53.374 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78043.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107517.log 2024-09-22T01:01:53.374 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40809.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40809.log.gz 2024-09-22T01:01:53.374 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69523.log 2024-09-22T01:01:53.374 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107517.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86686.log 2024-09-22T01:01:53.374 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69523.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37663.log 2024-09-22T01:01:53.374 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.86686.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45097.log.gz 2024-09-22T01:01:53.374 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45096.log.gz 2024-09-22T01:01:53.375 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116274.log 2024-09-22T01:01:53.375 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37663.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86563.log 2024-09-22T01:01:53.375 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116274.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138130.log 2024-09-22T01:01:53.375 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.86563.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138703.log 2024-09-22T01:01:53.375 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138130.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77285.log 2024-09-22T01:01:53.375 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75722.log 2024-09-22T01:01:53.375 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77285.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95287.log 2024-09-22T01:01:53.375 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91169.log 2024-09-22T01:01:53.375 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.95287.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32709.log 2024-09-22T01:01:53.375 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91169.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142503.log 2024-09-22T01:01:53.375 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32709.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138367.log 2024-09-22T01:01:53.375 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.142503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40833.log 2024-09-22T01:01:53.375 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.142503.log.gz 2024-09-22T01:01:53.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138367.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116779.log 2024-09-22T01:01:53.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.40833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136849.log 2024-09-22T01:01:53.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.116779.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.116779.log.gz 2024-09-22T01:01:53.376 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138574.log 2024-09-22T01:01:53.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136849.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125820.log 2024-09-22T01:01:53.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57295.log 2024-09-22T01:01:53.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59375.log 2024-09-22T01:01:53.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57295.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67844.log 2024-09-22T01:01:53.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126871.log 2024-09-22T01:01:53.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67844.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35248.log 2024-09-22T01:01:53.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126871.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32792.log 2024-09-22T01:01:53.376 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111825.log.gz 2024-09-22T01:01:53.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35248.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79177.log 2024-09-22T01:01:53.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93352.log 2024-09-22T01:01:53.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114090.log 2024-09-22T01:01:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93352.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81489.log 2024-09-22T01:01:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114090.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57821.log 2024-09-22T01:01:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81489.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.93480.log.gz 2024-09-22T01:01:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79430.log 2024-09-22T01:01:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130870.log 2024-09-22T01:01:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73538.log 2024-09-22T01:01:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.130870.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142006.log 2024-09-22T01:01:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73538.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32017.log 2024-09-22T01:01:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.142006.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105503.log 2024-09-22T01:01:53.377 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.142006.log.gz 2024-09-22T01:01:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32017.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40176.log 2024-09-22T01:01:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90960.log 2024-09-22T01:01:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40176.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47495.log 2024-09-22T01:01:53.377 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90960.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132238.log 2024-09-22T01:01:53.378 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47495.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82288.log 2024-09-22T01:01:53.378 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132238.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82940.log 2024-09-22T01:01:53.378 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31582.log 2024-09-22T01:01:53.378 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82940.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36975.log 2024-09-22T01:01:53.378 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31582.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.66876.log.gz 2024-09-22T01:01:53.378 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127476.log 2024-09-22T01:01:53.378 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36975.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66835.log.gz 2024-09-22T01:01:53.378 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93859.log 2024-09-22T01:01:53.378 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66877.log.gz 2024-09-22T01:01:53.378 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32413.log 2024-09-22T01:01:53.378 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93859.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59717.log 2024-09-22T01:01:53.378 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53751.log.gz 2024-09-22T01:01:53.378 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83685.log 2024-09-22T01:01:53.378 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70723.log 2024-09-22T01:01:53.378 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83685.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.54066.log.gz 2024-09-22T01:01:53.379 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83511.log 2024-09-22T01:01:53.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106688.log 2024-09-22T01:01:53.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67463.log 2024-09-22T01:01:53.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106688.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32668.log 2024-09-22T01:01:53.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67463.log: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54553.log.gz 2024-09-22T01:01:53.379 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65159.log 2024-09-22T01:01:53.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32668.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95890.log 2024-09-22T01:01:53.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91924.log 2024-09-22T01:01:53.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.95890.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129676.log 2024-09-22T01:01:53.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91924.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108997.log 2024-09-22T01:01:53.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78715.log 2024-09-22T01:01:53.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.108997.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58012.log 2024-09-22T01:01:53.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87362.log 2024-09-22T01:01:53.380 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56204.log 2024-09-22T01:01:53.380 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58473.log 2024-09-22T01:01:53.380 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56204.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29211.log 2024-09-22T01:01:53.380 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58473.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36470.log 2024-09-22T01:01:53.380 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29211.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29211.log.gz 2024-09-22T01:01:53.380 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75128.log 2024-09-22T01:01:53.380 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70932.log.gz 2024-09-22T01:01:53.380 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107255.log 2024-09-22T01:01:53.380 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36470.log: /var/log/ceph/cluster1-client.mirror.75128.log: 80.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41973.log.gz 2024-09-22T01:01:53.380 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60613.log 2024-09-22T01:01:53.380 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107255.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84503.log 2024-09-22T01:01:53.380 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.42031.log.gz 2024-09-22T01:01:53.380 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60613.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54340.log 2024-09-22T01:01:53.380 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116086.log 2024-09-22T01:01:53.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49804.log 2024-09-22T01:01:53.381 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138703.log.gz 2024-09-22T01:01:53.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116086.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34534.log 2024-09-22T01:01:53.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.49804.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.49804.log.gz 2024-09-22T01:01:53.381 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95712.log 2024-09-22T01:01:53.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34534.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75595.log 2024-09-22T01:01:53.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.95712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114660.log 2024-09-22T01:01:53.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75595.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138785.log 2024-09-22T01:01:53.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114660.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50942.log 2024-09-22T01:01:53.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90898.log 2024-09-22T01:01:53.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.50942.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85513.log 2024-09-22T01:01:53.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137103.log 2024-09-22T01:01:53.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.85513.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82690.log 2024-09-22T01:01:53.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137103.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39763.log 2024-09-22T01:01:53.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82690.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87572.log 2024-09-22T01:01:53.382 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39763.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92470.log 2024-09-22T01:01:53.382 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87572.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33821.log 2024-09-22T01:01:53.382 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92470.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127292.log.gz 2024-09-22T01:01:53.382 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127291.log.gz 2024-09-22T01:01:53.382 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67004.log 2024-09-22T01:01:53.382 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102220.log 2024-09-22T01:01:53.382 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92975.log 2024-09-22T01:01:53.382 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36680.log 2024-09-22T01:01:53.382 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92975.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130984.log 2024-09-22T01:01:53.382 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36680.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126912.log 2024-09-22T01:01:53.382 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.130984.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55702.log 2024-09-22T01:01:53.382 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126912.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108541.log 2024-09-22T01:01:53.382 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69042.log 2024-09-22T01:01:53.382 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.108541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128718.log 2024-09-22T01:01:53.383 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69042.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100542.log 2024-09-22T01:01:53.383 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128718.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106118.log 2024-09-22T01:01:53.383 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63312.log 2024-09-22T01:01:53.383 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.74091.log.gz 2024-09-22T01:01:53.383 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106118.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89782.log 2024-09-22T01:01:53.383 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139191.log 2024-09-22T01:01:53.383 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126323.log 2024-09-22T01:01:53.383 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139191.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.139191.log.gz 2024-09-22T01:01:53.383 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112937.log 2024-09-22T01:01:53.383 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120736.log 2024-09-22T01:01:53.383 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112937.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118624.log 2024-09-22T01:01:53.383 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129896.log 2024-09-22T01:01:53.383 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.118624.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80691.log 2024-09-22T01:01:53.384 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.129896.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49740.log 2024-09-22T01:01:53.384 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84332.log.gz 2024-09-22T01:01:53.384 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80691.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49009.log 2024-09-22T01:01:53.384 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.49740.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.49740.log.gz 2024-09-22T01:01:53.384 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68598.log 2024-09-22T01:01:53.384 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142522.log 2024-09-22T01:01:53.384 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68598.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84483.log 2024-09-22T01:01:53.384 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.142522.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120480.log 2024-09-22T01:01:53.384 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.142522.log.gz 2024-09-22T01:01:53.384 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105696.log 2024-09-22T01:01:53.384 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120480.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35351.log 2024-09-22T01:01:53.384 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105696.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46992.log 2024-09-22T01:01:53.384 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32203.log 2024-09-22T01:01:53.384 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92850.log 2024-09-22T01:01:53.384 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32203.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118306.log 2024-09-22T01:01:53.385 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61323.log 2024-09-22T01:01:53.385 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.118306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34789.log 2024-09-22T01:01:53.385 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125484.log 2024-09-22T01:01:53.385 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34789.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56752.log 2024-09-22T01:01:53.385 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125484.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80061.log 2024-09-22T01:01:53.385 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.104257.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.104256.log.gz 2024-09-22T01:01:53.385 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:01:53.385 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139894.log 2024-09-22T01:01:53.385 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.107255.log.gz 2024-09-22T01:01:53.385 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139715.log 2024-09-22T01:01:53.385 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122962.log.gz 2024-09-22T01:01:53.385 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136388.log 2024-09-22T01:01:53.385 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119948.log 2024-09-22T01:01:53.385 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136388.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.128571.log.gz 2024-09-22T01:01:53.386 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135631.log 2024-09-22T01:01:53.386 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.119948.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112938.log 2024-09-22T01:01:53.386 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135631.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47454.log 2024-09-22T01:01:53.386 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112938.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122963.log 2024-09-22T01:01:53.386 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31948.log 2024-09-22T01:01:53.386 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60614.log 2024-09-22T01:01:53.386 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115352.log.gz 2024-09-22T01:01:53.386 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31948.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48504.log 2024-09-22T01:01:53.386 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60614.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.140065.log 2024-09-22T01:01:53.386 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48504.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57211.log 2024-09-22T01:01:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.140065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32331.log 2024-09-22T01:01:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57211.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42900.log 2024-09-22T01:01:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127499.log 2024-09-22T01:01:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42900.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99693.log 2024-09-22T01:01:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127499.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87399.log 2024-09-22T01:01:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99693.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43132.log 2024-09-22T01:01:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30524.log 2024-09-22T01:01:53.387 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72289.log.gz 2024-09-22T01:01:53.387 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100227.log.gz 2024-09-22T01:01:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78256.log 2024-09-22T01:01:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30524.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127646.log 2024-09-22T01:01:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52217.log 2024-09-22T01:01:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127646.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60762.log 2024-09-22T01:01:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.52217.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108039.log 2024-09-22T01:01:53.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60762.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115508.log 2024-09-22T01:01:53.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.108039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113460.log 2024-09-22T01:01:53.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115508.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133748.log 2024-09-22T01:01:53.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69461.log 2024-09-22T01:01:53.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133748.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33610.log 2024-09-22T01:01:53.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69461.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88137.log 2024-09-22T01:01:53.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59018.log 2024-09-22T01:01:53.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88137.log: 80.6% -- replaced with /var/log/ceph/cluster1-client.mirror.132238.log.gz 2024-09-22T01:01:53.388 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137525.log 2024-09-22T01:01:53.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113251.log 2024-09-22T01:01:53.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137525.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104887.log 2024-09-22T01:01:53.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102012.log 2024-09-22T01:01:53.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33087.log 2024-09-22T01:01:53.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102012.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118139.log 2024-09-22T01:01:53.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33087.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132388.log 2024-09-22T01:01:53.389 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118139.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103836.log 2024-09-22T01:01:53.389 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132388.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43322.log 2024-09-22T01:01:53.389 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103836.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37791.log 2024-09-22T01:01:53.389 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.43322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34369.log 2024-09-22T01:01:53.389 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33043.log 2024-09-22T01:01:53.389 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55784.log 2024-09-22T01:01:53.389 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114804.log 2024-09-22T01:01:53.389 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139262.log.gz/var/log/ceph/cluster1-client.mirror.55784.log: 2024-09-22T01:01:53.389 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141819.log 2024-09-22T01:01:53.389 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114804.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35922.log 2024-09-22T01:01:53.389 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.141819.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39614.log 2024-09-22T01:01:53.389 INFO:teuthology.orchestra.run.smithi060.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.141819.log.gz 2024-09-22T01:01:53.389 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35922.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90688.log 2024-09-22T01:01:53.389 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39614.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106586.log 2024-09-22T01:01:53.389 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90688.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130756.log 2024-09-22T01:01:53.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106586.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60235.log 2024-09-22T01:01:53.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.130756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96981.log 2024-09-22T01:01:53.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60235.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76331.log 2024-09-22T01:01:53.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.96981.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118329.log 2024-09-22T01:01:53.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102957.log 2024-09-22T01:01:53.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.118329.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101800.log 2024-09-22T01:01:53.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102957.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33002.log 2024-09-22T01:01:53.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87424.log 2024-09-22T01:01:53.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90540.log 2024-09-22T01:01:53.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87424.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83147.log 2024-09-22T01:01:53.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56498.log 2024-09-22T01:01:53.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83147.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.83147.log.gz 2024-09-22T01:01:53.390 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87434.log 2024-09-22T01:01:53.390 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104929.log.gz 2024-09-22T01:01:53.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34410.log 2024-09-22T01:01:53.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87434.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66367.log 2024-09-22T01:01:53.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34410.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55825.log 2024-09-22T01:01:53.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66367.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119264.log 2024-09-22T01:01:53.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55825.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80019.log 2024-09-22T01:01:53.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.119264.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40358.log 2024-09-22T01:01:53.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127542.log 2024-09-22T01:01:53.391 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2%/var/log/ceph/cluster2-client.admin.40358.log: -- replaced with /var/log/ceph/cluster1-client.mirror.105013.log.gz 2024-09-22T01:01:53.391 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124746.log 2024-09-22T01:01:53.391 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.105056.log.gz 2024-09-22T01:01:53.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32833.log 2024-09-22T01:01:53.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113779.log 2024-09-22T01:01:53.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63701.log 2024-09-22T01:01:53.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113779.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44804.log 2024-09-22T01:01:53.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.63701.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101862.log 2024-09-22T01:01:53.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44804.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33318.log 2024-09-22T01:01:53.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101862.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70153.log 2024-09-22T01:01:53.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33318.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131496.log 2024-09-22T01:01:53.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70153.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66667.log 2024-09-22T01:01:53.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131496.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137060.log 2024-09-22T01:01:53.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64043.log 2024-09-22T01:01:53.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137060.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124706.log 2024-09-22T01:01:53.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.64043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137702.log 2024-09-22T01:01:53.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124706.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125443.log 2024-09-22T01:01:53.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103901.log 2024-09-22T01:01:53.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33529.log 2024-09-22T01:01:53.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103901.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113821.log 2024-09-22T01:01:53.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33529.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65905.log 2024-09-22T01:01:53.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104427.log 2024-09-22T01:01:53.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116963.log 2024-09-22T01:01:53.393 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43391.log.gz 2024-09-22T01:01:53.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104427.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133091.log 2024-09-22T01:01:53.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70428.log 2024-09-22T01:01:53.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114031.log 2024-09-22T01:01:53.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70428.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112661.log 2024-09-22T01:01:53.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114031.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96819.log 2024-09-22T01:01:53.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120088.log 2024-09-22T01:01:53.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96819.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96819.log.gz 2024-09-22T01:01:53.393 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47035.log 2024-09-22T01:01:53.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.120088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130548.log 2024-09-22T01:01:53.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47035.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114552.log 2024-09-22T01:01:53.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.130548.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51408.log 2024-09-22T01:01:53.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50440.log 2024-09-22T01:01:53.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51408.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109948.log 2024-09-22T01:01:53.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50440.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80399.log 2024-09-22T01:01:53.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109948.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37098.log 2024-09-22T01:01:53.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80399.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84389.log 2024-09-22T01:01:53.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37098.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93962.log 2024-09-22T01:01:53.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38919.log 2024-09-22T01:01:53.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93962.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79555.log 2024-09-22T01:01:53.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97189.log 2024-09-22T01:01:53.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79555.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76564.log 2024-09-22T01:01:53.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.97189.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52813.log 2024-09-22T01:01:53.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82540.log 2024-09-22T01:01:53.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52813.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63218.log 2024-09-22T01:01:53.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67297.log 2024-09-22T01:01:53.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.63218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89699.log 2024-09-22T01:01:53.395 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67297.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126662.log 2024-09-22T01:01:53.395 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89699.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50208.log 2024-09-22T01:01:53.395 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126662.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141497.log 2024-09-22T01:01:53.395 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136680.log 2024-09-22T01:01:53.395 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.141497.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88790.log 2024-09-22T01:01:53.395 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136680.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.141497.log.gz 2024-09-22T01:01:53.395 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29219.log 2024-09-22T01:01:53.395 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88790.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141287.log 2024-09-22T01:01:53.395 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29219.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29219.log.gz 2024-09-22T01:01:53.395 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139773.log 2024-09-22T01:01:53.395 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.141287.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141287.log.gz 2024-09-22T01:01:53.395 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58726.log 2024-09-22T01:01:53.395 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139773.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131723.log 2024-09-22T01:01:53.395 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58726.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111931.log 2024-09-22T01:01:53.395 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135651.log 2024-09-22T01:01:53.396 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125586.log 2024-09-22T01:01:53.396 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103691.log 2024-09-22T01:01:53.396 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125586.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99293.log 2024-09-22T01:01:53.396 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103691.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31946.log 2024-09-22T01:01:53.396 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99293.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134664.log 2024-09-22T01:01:53.396 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31946.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113039.log 2024-09-22T01:01:53.396 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134664.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39618.log 2024-09-22T01:01:53.396 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57043.log 2024-09-22T01:01:53.396 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39618.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41343.log 2024-09-22T01:01:53.396 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106366.log 2024-09-22T01:01:53.396 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41343.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82919.log 2024-09-22T01:01:53.396 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36429.log 2024-09-22T01:01:53.396 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136849.log.gz 2024-09-22T01:01:53.396 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82919.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84781.log 2024-09-22T01:01:53.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56077.log 2024-09-22T01:01:53.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84781.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79723.log 2024-09-22T01:01:53.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56077.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96365.log 2024-09-22T01:01:53.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44130.log 2024-09-22T01:01:53.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.96365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34932.log 2024-09-22T01:01:53.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44130.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35208.log 2024-09-22T01:01:53.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34932.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103543.log.gz 2024-09-22T01:01:53.397 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136511.log 2024-09-22T01:01:53.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99425.log 2024-09-22T01:01:53.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100646.log 2024-09-22T01:01:53.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99425.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95345.log 2024-09-22T01:01:53.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100646.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37769.log 2024-09-22T01:01:53.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.95345.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49329.log 2024-09-22T01:01:53.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37769.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140915.log 2024-09-22T01:01:53.398 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49329.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81786.log 2024-09-22T01:01:53.398 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140915.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86086.log 2024-09-22T01:01:53.398 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39931.log 2024-09-22T01:01:53.398 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.86086.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62747.log 2024-09-22T01:01:53.398 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39931.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62422.log 2024-09-22T01:01:53.398 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62747.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53557.log 2024-09-22T01:01:53.398 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.62422.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56708.log 2024-09-22T01:01:53.398 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53557.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28854.log 2024-09-22T01:01:53.398 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56708.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140698.log 2024-09-22T01:01:53.398 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.28854.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28854.log.gz 2024-09-22T01:01:53.398 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.109467.log.gz 2024-09-22T01:01:53.398 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84442.log 2024-09-22T01:01:53.398 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.140698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35878.log 2024-09-22T01:01:53.398 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77834.log 2024-09-22T01:01:53.399 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35878.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42985.log 2024-09-22T01:01:53.399 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96018.log 2024-09-22T01:01:53.399 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72202.log 2024-09-22T01:01:53.399 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.96018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70827.log 2024-09-22T01:01:53.399 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72202.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.72202.log.gz 2024-09-22T01:01:53.399 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69438.log 2024-09-22T01:01:53.399 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70827.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71623.log 2024-09-22T01:01:53.399 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69438.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63112.log 2024-09-22T01:01:53.399 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71623.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76306.log 2024-09-22T01:01:53.399 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.63112.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63112.log.gz 2024-09-22T01:01:53.399 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125610.log 2024-09-22T01:01:53.399 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47982.log 2024-09-22T01:01:53.399 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55182.log.gz 2024-09-22T01:01:53.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67371.log 2024-09-22T01:01:53.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47982.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42837.log 2024-09-22T01:01:53.400 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.94814.log.gz 2024-09-22T01:01:53.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67371.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121729.log 2024-09-22T01:01:53.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42837.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79936.log 2024-09-22T01:01:53.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121729.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116442.log 2024-09-22T01:01:53.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79936.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91441.log 2024-09-22T01:01:53.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136721.log 2024-09-22T01:01:53.400 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.95287.log.gz 2024-09-22T01:01:53.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91441.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95275.log.gz 2024-09-22T01:01:53.400 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96386.log 2024-09-22T01:01:53.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136721.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78799.log 2024-09-22T01:01:53.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96386.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81951.log 2024-09-22T01:01:53.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78799.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.76306.log.gz 2024-09-22T01:01:53.400 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133949.log 2024-09-22T01:01:53.401 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81951.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100285.log 2024-09-22T01:01:53.401 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133949.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.98275.log.gz 2024-09-22T01:01:53.401 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82077.log 2024-09-22T01:01:53.401 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100285.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96247.log 2024-09-22T01:01:53.401 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82077.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107089.log 2024-09-22T01:01:53.401 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.96247.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114877.log 2024-09-22T01:01:53.401 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.107089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138156.log 2024-09-22T01:01:53.401 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69254.log 2024-09-22T01:01:53.401 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133364.log 2024-09-22T01:01:53.401 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69254.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38635.log 2024-09-22T01:01:53.401 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133364.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100098.log 2024-09-22T01:01:53.401 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93859.log.gz 2024-09-22T01:01:53.401 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125862.log 2024-09-22T01:01:53.401 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100098.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94309.log 2024-09-22T01:01:53.402 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125862.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46398.log 2024-09-22T01:01:53.402 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45267.log 2024-09-22T01:01:53.402 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91748.log 2024-09-22T01:01:53.402 INFO:teuthology.orchestra.run.smithi060.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75128.log.gz 2024-09-22T01:01:53.402 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45267.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60236.log 2024-09-22T01:01:53.402 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91748.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123385.log 2024-09-22T01:01:53.402 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60236.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90454.log 2024-09-22T01:01:53.402 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123385.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38693.log 2024-09-22T01:01:53.402 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41343.log.gz 2024-09-22T01:01:53.402 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112454.log 2024-09-22T01:01:53.402 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38693.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86449.log 2024-09-22T01:01:53.402 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138364.log 2024-09-22T01:01:53.402 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42394.log.gz 2024-09-22T01:01:53.510 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.86449.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125344.log 2024-09-22T01:01:53.511 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60978.log.gz 2024-09-22T01:01:53.511 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138364.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60977.log.gz 2024-09-22T01:01:53.511 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31351.log 2024-09-22T01:01:53.511 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125344.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75148.log 2024-09-22T01:01:53.511 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38694.log 2024-09-22T01:01:53.511 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.75148.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56436.log 2024-09-22T01:01:53.511 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38694.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67506.log 2024-09-22T01:01:53.511 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56436.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98971.log 2024-09-22T01:01:53.511 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67506.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82120.log 2024-09-22T01:01:53.511 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126426.log 2024-09-22T01:01:53.511 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.51395.log.gz 2024-09-22T01:01:53.511 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82120.log: gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-09-22T01:01:53.511 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83836.log 2024-09-22T01:01:53.512 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139218.log 2024-09-22T01:01:53.512 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64183.log.gz 2024-09-22T01:01:53.512 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83836.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47538.log 2024-09-22T01:01:53.512 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75429.log.gz 2024-09-22T01:01:53.512 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139218.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.139218.log.gz 2024-09-22T01:01:53.512 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67296.log 2024-09-22T01:01:53.512 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47538.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49415.log 2024-09-22T01:01:53.512 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.119921.log.gz 2024-09-22T01:01:53.512 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119974.log.gz 2024-09-22T01:01:53.512 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67296.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100478.log 2024-09-22T01:01:53.512 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49415.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101358.log 2024-09-22T01:01:53.512 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100478.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69254.log.gz 2024-09-22T01:01:53.512 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95126.log 2024-09-22T01:01:53.512 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101358.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60278.log 2024-09-22T01:01:53.512 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.95126.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59186.log 2024-09-22T01:01:53.513 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60278.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76121.log 2024-09-22T01:01:53.513 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59186.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74765.log 2024-09-22T01:01:53.513 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59482.log 2024-09-22T01:01:53.513 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.74765.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103481.log 2024-09-22T01:01:53.513 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114131.log 2024-09-22T01:01:53.513 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103481.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81156.log 2024-09-22T01:01:53.513 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63245.log 2024-09-22T01:01:53.513 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97531.log 2024-09-22T01:01:53.513 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.63245.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122308.log 2024-09-22T01:01:53.513 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.97531.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128529.log 2024-09-22T01:01:53.513 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122308.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120819.log 2024-09-22T01:01:53.513 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128529.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123383.log 2024-09-22T01:01:53.513 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.120819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30866.log 2024-09-22T01:01:53.513 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90749.log 2024-09-22T01:01:53.514 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30866.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67968.log 2024-09-22T01:01:53.514 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90749.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128246.log 2024-09-22T01:01:53.514 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67968.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128172.log 2024-09-22T01:01:53.514 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.128246.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57463.log 2024-09-22T01:01:53.514 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.128172.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51104.log 2024-09-22T01:01:53.514 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57463.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127900.log 2024-09-22T01:01:53.514 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51104.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124014.log 2024-09-22T01:01:53.514 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127900.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118389.log 2024-09-22T01:01:53.514 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124014.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126533.log 2024-09-22T01:01:53.514 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104173.log 2024-09-22T01:01:53.514 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126533.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90183.log 2024-09-22T01:01:53.514 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104173.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103694.log 2024-09-22T01:01:53.514 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59017.log 2024-09-22T01:01:53.514 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103694.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58894.log 2024-09-22T01:01:53.515 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59017.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64499.log 2024-09-22T01:01:53.515 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58766.log.gz 2024-09-22T01:01:53.515 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115182.log 2024-09-22T01:01:53.515 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64499.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32477.log 2024-09-22T01:01:53.515 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65538.log 2024-09-22T01:01:53.515 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114618.log 2024-09-22T01:01:53.515 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65538.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129246.log 2024-09-22T01:01:53.515 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114618.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120861.log 2024-09-22T01:01:53.515 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129246.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30665.log 2024-09-22T01:01:53.515 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120861.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132302.log 2024-09-22T01:01:53.515 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30665.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58891.log.gz 2024-09-22T01:01:53.515 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119331.log 2024-09-22T01:01:53.515 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58850.log.gz 2024-09-22T01:01:53.515 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132302.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101526.log 2024-09-22T01:01:53.515 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.119331.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140757.log 2024-09-22T01:01:53.516 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101526.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36301.log 2024-09-22T01:01:53.516 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93352.log.gz 2024-09-22T01:01:53.516 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138322.log 2024-09-22T01:01:53.516 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36301.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59375.log.gz 2024-09-22T01:01:53.516 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119880.log 2024-09-22T01:01:53.516 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138322.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108521.log 2024-09-22T01:01:53.516 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.119880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105164.log 2024-09-22T01:01:53.516 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.108521.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115603.log 2024-09-22T01:01:53.516 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105164.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55531.log 2024-09-22T01:01:53.516 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87930.log 2024-09-22T01:01:53.516 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55531.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137524.log 2024-09-22T01:01:53.516 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.87930.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88717.log.gz 2024-09-22T01:01:53.516 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123446.log 2024-09-22T01:01:53.516 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137524.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137186.log 2024-09-22T01:01:53.517 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123446.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135652.log 2024-09-22T01:01:53.517 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137186.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109573.log 2024-09-22T01:01:53.517 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34368.log 2024-09-22T01:01:53.517 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109573.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132720.log 2024-09-22T01:01:53.517 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34368.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103694.log.gz 2024-09-22T01:01:53.517 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32960.log 2024-09-22T01:01:53.517 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132720.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140811.log 2024-09-22T01:01:53.517 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32960.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72919.log 2024-09-22T01:01:53.517 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140811.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115038.log 2024-09-22T01:01:53.517 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72919.log: gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-09-22T01:01:53.517 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115038.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94036.log 2024-09-22T01:01:53.517 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74241.log 2024-09-22T01:01:53.517 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94036.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69144.log 2024-09-22T01:01:53.518 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.74241.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74241.log.gz 2024-09-22T01:01:53.518 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54720.log 2024-09-22T01:01:53.518 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69144.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81700.log 2024-09-22T01:01:53.518 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.81951.log.gz 2024-09-22T01:01:53.518 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83106.log 2024-09-22T01:01:53.518 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51265.log 2024-09-22T01:01:53.518 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.83106.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80987.log 2024-09-22T01:01:53.518 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.51265.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57675.log 2024-09-22T01:01:53.518 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80987.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33487.log 2024-09-22T01:01:53.518 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57675.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47792.log 2024-09-22T01:01:53.518 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116402.log 2024-09-22T01:01:53.518 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113356.log 2024-09-22T01:01:53.518 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116402.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64244.log 2024-09-22T01:01:53.518 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113356.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41887.log 2024-09-22T01:01:53.518 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.64244.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103962.log 2024-09-22T01:01:53.519 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.41887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64016.log 2024-09-22T01:01:53.519 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103962.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87424.log.gz 2024-09-22T01:01:53.519 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64541.log 2024-09-22T01:01:53.519 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.64016.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69858.log 2024-09-22T01:01:53.519 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.64541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136032.log 2024-09-22T01:01:53.519 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69858.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92220.log 2024-09-22T01:01:53.519 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136032.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52178.log 2024-09-22T01:01:53.519 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75849.log 2024-09-22T01:01:53.519 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster2.log.gz 2024-09-22T01:01:53.519 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.52178.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52178.log.gz 2024-09-22T01:01:53.519 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101316.log 2024-09-22T01:01:53.519 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75849.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52418.log 2024-09-22T01:01:53.519 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101316.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121706.log 2024-09-22T01:01:53.519 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.52418.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52901.log 2024-09-22T01:01:53.520 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.121706.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131171.log 2024-09-22T01:01:53.520 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.52901.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61058.log 2024-09-22T01:01:53.520 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.131171.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71666.log 2024-09-22T01:01:53.520 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61058.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110697.log 2024-09-22T01:01:53.520 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71666.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80901.log 2024-09-22T01:01:53.520 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110697.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92850.log.gz 2024-09-22T01:01:53.520 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52284.log 2024-09-22T01:01:53.520 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80901.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106637.log 2024-09-22T01:01:53.520 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52284.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114449.log 2024-09-22T01:01:53.520 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34998.log 2024-09-22T01:01:53.520 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114449.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95708.log 2024-09-22T01:01:53.520 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34998.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59604.log 2024-09-22T01:01:53.520 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.95708.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140691.log 2024-09-22T01:01:53.520 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59604.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110757.log.gz 2024-09-22T01:01:53.521 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95127.log 2024-09-22T01:01:53.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.140691.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56834.log 2024-09-22T01:01:53.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.95127.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129549.log 2024-09-22T01:01:53.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56834.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112661.log.gz 2024-09-22T01:01:53.521 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44427.log 2024-09-22T01:01:53.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129549.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102852.log 2024-09-22T01:01:53.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44427.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81155.log 2024-09-22T01:01:53.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102852.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83267.log 2024-09-22T01:01:53.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81155.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31945.log 2024-09-22T01:01:53.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.83267.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121877.log 2024-09-22T01:01:53.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33822.log 2024-09-22T01:01:53.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132921.log 2024-09-22T01:01:53.521 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134622.log.gz 2024-09-22T01:01:53.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33822.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108790.log 2024-09-22T01:01:53.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132921.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111508.log 2024-09-22T01:01:53.522 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.108790.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31138.log 2024-09-22T01:01:53.522 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111508.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84571.log 2024-09-22T01:01:53.522 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76880.log 2024-09-22T01:01:53.522 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.84571.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128193.log 2024-09-22T01:01:53.522 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92788.log 2024-09-22T01:01:53.522 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128193.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128193.log.gz 2024-09-22T01:01:53.522 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76459.log 2024-09-22T01:01:53.522 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113356.log.gz 2024-09-22T01:01:53.522 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92788.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32113.log 2024-09-22T01:01:53.522 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76459.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86898.log 2024-09-22T01:01:53.522 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32113.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137863.log 2024-09-22T01:01:53.522 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.86898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35082.log 2024-09-22T01:01:53.522 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137863.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122308.log.gz 2024-09-22T01:01:53.523 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59858.log 2024-09-22T01:01:53.523 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35082.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39138.log 2024-09-22T01:01:53.523 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59858.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57253.log 2024-09-22T01:01:53.523 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122755.log 2024-09-22T01:01:53.523 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57253.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113568.log 2024-09-22T01:01:53.523 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122755.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89221.log 2024-09-22T01:01:53.523 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113568.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103272.log 2024-09-22T01:01:53.523 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56371.log 2024-09-22T01:01:53.523 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55183.log 2024-09-22T01:01:53.523 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56371.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70783.log 2024-09-22T01:01:53.523 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53962.log 2024-09-22T01:01:53.523 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70783.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42136.log 2024-09-22T01:01:53.523 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53962.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126031.log 2024-09-22T01:01:53.523 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53645.log 2024-09-22T01:01:53.523 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81415.log 2024-09-22T01:01:53.524 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112938.log.gz 2024-09-22T01:01:53.524 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.53645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41518.log 2024-09-22T01:01:53.524 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81415.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56331.log 2024-09-22T01:01:53.524 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.41518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100204.log 2024-09-22T01:01:53.524 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103123.log 2024-09-22T01:01:53.524 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126597.log.gz 2024-09-22T01:01:53.524 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100204.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93730.log 2024-09-22T01:01:53.524 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40886.log 2024-09-22T01:01:53.524 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93730.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105434.log 2024-09-22T01:01:53.524 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59521.log 2024-09-22T01:01:53.524 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105434.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120925.log 2024-09-22T01:01:53.524 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59521.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104741.log 2024-09-22T01:01:53.524 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120925.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63385.log 2024-09-22T01:01:53.524 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77669.log 2024-09-22T01:01:53.525 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.63385.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78378.log 2024-09-22T01:01:53.525 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77669.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83797.log 2024-09-22T01:01:53.525 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78378.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115603.log.gz 2024-09-22T01:01:53.525 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98554.log 2024-09-22T01:01:53.525 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83797.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125546.log 2024-09-22T01:01:53.525 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97738.log 2024-09-22T01:01:53.525 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125546.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41726.log 2024-09-22T01:01:53.525 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.97738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54746.log 2024-09-22T01:01:53.525 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41726.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71033.log 2024-09-22T01:01:53.525 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49477.log 2024-09-22T01:01:53.525 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123383.log.gz 2024-09-22T01:01:53.525 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123385.log.gz 2024-09-22T01:01:53.525 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83432.log 2024-09-22T01:01:53.525 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79557.log 2024-09-22T01:01:53.526 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83432.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108723.log 2024-09-22T01:01:53.526 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79557.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68933.log 2024-09-22T01:01:53.526 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.108723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70236.log 2024-09-22T01:01:53.526 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68933.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44300.log 2024-09-22T01:01:53.526 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70236.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107976.log 2024-09-22T01:01:53.526 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.83836.log.gz 2024-09-22T01:01:53.526 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44300.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99693.log.gz 2024-09-22T01:01:53.526 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30602.log 2024-09-22T01:01:53.526 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107976.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.107976.log.gz 2024-09-22T01:01:53.526 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135041.log 2024-09-22T01:01:53.526 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49094.log 2024-09-22T01:01:53.526 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135041.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141638.log 2024-09-22T01:01:53.526 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49094.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54956.log 2024-09-22T01:01:53.527 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.141638.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57381.log 2024-09-22T01:01:53.527 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54956.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127178.log.gz 2024-09-22T01:01:53.527 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52140.log 2024-09-22T01:01:53.527 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57381.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34304.log 2024-09-22T01:01:53.527 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.52140.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52140.log.gz 2024-09-22T01:01:53.527 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115395.log 2024-09-22T01:01:53.527 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34304.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.141638.log.gz 2024-09-22T01:01:53.527 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100351.log 2024-09-22T01:01:53.527 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115395.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.84503.log.gz 2024-09-22T01:01:53.527 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31390.log 2024-09-22T01:01:53.527 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100351.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25405.log 2024-09-22T01:01:53.527 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84571.log.gz 2024-09-22T01:01:53.527 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.31390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63426.log 2024-09-22T01:01:53.528 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.132302.log.gz 2024-09-22T01:01:53.528 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.25405.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51080.log 2024-09-22T01:01:53.528 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88041.log 2024-09-22T01:01:53.528 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51080.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76796.log 2024-09-22T01:01:53.528 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88041.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136178.log 2024-09-22T01:01:53.528 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76796.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107256.log 2024-09-22T01:01:53.528 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136178.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93103.log 2024-09-22T01:01:53.528 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97551.log 2024-09-22T01:01:53.528 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98492.log 2024-09-22T01:01:53.528 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.97551.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75720.log 2024-09-22T01:01:53.528 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98492.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107243.log 2024-09-22T01:01:53.528 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136979.log.gz 2024-09-22T01:01:53.528 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99965.log 2024-09-22T01:01:53.528 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107243.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.109573.log.gz 2024-09-22T01:01:53.529 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108635.log 2024-09-22T01:01:53.529 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99965.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35839.log 2024-09-22T01:01:53.529 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.108635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68347.log 2024-09-22T01:01:53.529 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35839.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103174.log 2024-09-22T01:01:53.529 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112454.log.gz 2024-09-22T01:01:53.529 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68347.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42963.log 2024-09-22T01:01:53.529 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114971.log 2024-09-22T01:01:53.529 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87742.log 2024-09-22T01:01:53.529 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60068.log 2024-09-22T01:01:53.529 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104320.log 2024-09-22T01:01:53.529 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60068.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79936.log.gz 2024-09-22T01:01:53.529 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117095.log 2024-09-22T01:01:53.529 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104320.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85838.log 2024-09-22T01:01:53.529 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.117095.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.117095.log.gz 2024-09-22T01:01:53.529 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124326.log 2024-09-22T01:01:53.530 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.85838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90750.log 2024-09-22T01:01:53.530 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124326.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46316.log 2024-09-22T01:01:53.530 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90750.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.80019.log.gz 2024-09-22T01:01:53.530 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80061.log.gz 2024-09-22T01:01:53.530 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81699.log 2024-09-22T01:01:53.530 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46316.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40106.log 2024-09-22T01:01:53.530 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81699.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39728.log 2024-09-22T01:01:53.530 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40106.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120671.log 2024-09-22T01:01:53.530 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39728.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54554.log 2024-09-22T01:01:53.530 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120671.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30687.log 2024-09-22T01:01:53.530 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54554.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138784.log.gz 2024-09-22T01:01:53.530 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36639.log 2024-09-22T01:01:53.530 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.30687.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97396.log 2024-09-22T01:01:53.531 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36639.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115601.log 2024-09-22T01:01:53.531 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.97396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45852.log 2024-09-22T01:01:53.531 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117833.log 2024-09-22T01:01:53.531 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45852.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34619.log 2024-09-22T01:01:53.531 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.117833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64584.log 2024-09-22T01:01:53.531 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34619.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124391.log 2024-09-22T01:01:53.531 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64584.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55783.log 2024-09-22T01:01:53.531 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136388.log.gz 2024-09-22T01:01:53.531 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141684.log 2024-09-22T01:01:53.531 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55783.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31834.log 2024-09-22T01:01:53.531 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.141684.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127436.log 2024-09-22T01:01:53.531 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31834.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129413.log 2024-09-22T01:01:53.531 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127436.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32877.log 2024-09-22T01:01:53.531 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.129413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37141.log 2024-09-22T01:01:53.531 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97122.log 2024-09-22T01:01:53.532 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37141.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56920.log 2024-09-22T01:01:53.532 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.97122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133048.log 2024-09-22T01:01:53.532 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56920.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62223.log 2024-09-22T01:01:53.532 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133048.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134999.log 2024-09-22T01:01:53.532 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.97551.log.gz 2024-09-22T01:01:53.532 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62223.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50439.log 2024-09-22T01:01:53.532 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56708.log.gz 2024-09-22T01:01:53.532 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134999.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45851.log 2024-09-22T01:01:53.532 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50439.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131819.log 2024-09-22T01:01:53.532 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45851.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141684.log.gz 2024-09-22T01:01:53.532 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130393.log 2024-09-22T01:01:53.532 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46614.log 2024-09-22T01:01:53.532 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.130393.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50817.log 2024-09-22T01:01:53.532 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46614.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92580.log 2024-09-22T01:01:53.533 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.50817.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106776.log 2024-09-22T01:01:53.533 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92580.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47982.log.gz 2024-09-22T01:01:53.533 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52166.log 2024-09-22T01:01:53.533 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.106776.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97578.log 2024-09-22T01:01:53.533 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.52166.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52166.log.gz 2024-09-22T01:01:53.533 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55029.log 2024-09-22T01:01:53.533 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.97578.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134665.log 2024-09-22T01:01:53.533 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90477.log 2024-09-22T01:01:53.533 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134665.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102011.log 2024-09-22T01:01:53.533 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43234.log 2024-09-22T01:01:53.533 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102011.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119901.log 2024-09-22T01:01:53.533 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43234.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29641.log 2024-09-22T01:01:53.533 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.119901.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75932.log 2024-09-22T01:01:53.533 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29641.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29641.log.gz 2024-09-22T01:01:53.534 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 87.7% --verbose -- -- replaced with /var/log/ceph/cluster1-client.mirror.131496.log.gz /var/log/ceph/cluster1-client.mirror.102072.log 2024-09-22T01:01:53.534 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:01:53.534 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.131517.log.gz 2024-09-22T01:01:53.534 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.75932.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101802.log 2024-09-22T01:01:53.534 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56665.log 2024-09-22T01:01:53.534 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47983.log 2024-09-22T01:01:53.534 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56665.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104090.log 2024-09-22T01:01:53.534 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47983.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89929.log 2024-09-22T01:01:53.534 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104090.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42331.log 2024-09-22T01:01:53.534 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52377.log 2024-09-22T01:01:53.534 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53688.log 2024-09-22T01:01:53.534 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52377.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133219.log 2024-09-22T01:01:53.534 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53688.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77102.log 2024-09-22T01:01:53.534 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83265.log 2024-09-22T01:01:53.535 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.133091.log.gz 2024-09-22T01:01:53.535 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77102.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53878.log 2024-09-22T01:01:53.535 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.83265.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41773.log 2024-09-22T01:01:53.535 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84737.log 2024-09-22T01:01:53.535 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.41773.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98934.log 2024-09-22T01:01:53.535 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84737.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38089.log 2024-09-22T01:01:53.535 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98934.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84280.log 2024-09-22T01:01:53.535 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.38089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45600.log 2024-09-22T01:01:53.535 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84280.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136031.log 2024-09-22T01:01:53.535 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45600.log: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.86449.log.gz 2024-09-22T01:01:53.535 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37479.log 2024-09-22T01:01:53.535 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42521.log 2024-09-22T01:01:53.535 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70513.log 2024-09-22T01:01:53.536 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42521.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41203.log 2024-09-22T01:01:53.536 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70513.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139921.log 2024-09-22T01:01:53.536 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.41203.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99635.log 2024-09-22T01:01:53.536 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139921.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48044.log 2024-09-22T01:01:53.536 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135252.log 2024-09-22T01:01:53.536 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45267.log.gz 2024-09-22T01:01:53.536 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109255.log 2024-09-22T01:01:53.536 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135252.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95193.log 2024-09-22T01:01:53.536 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109255.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74464.log 2024-09-22T01:01:53.536 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.95193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104740.log 2024-09-22T01:01:53.536 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.74464.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96042.log 2024-09-22T01:01:53.536 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101277.log 2024-09-22T01:01:53.536 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96042.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80106.log 2024-09-22T01:01:53.536 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101277.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33253.log 2024-09-22T01:01:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80106.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80902.log 2024-09-22T01:01:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33253.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124013.log 2024-09-22T01:01:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37520.log 2024-09-22T01:01:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124013.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67574.log 2024-09-22T01:01:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37520.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49051.log 2024-09-22T01:01:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30504.log 2024-09-22T01:01:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49051.log: /var/log/ceph/cluster1-client.mirror.30504.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78614.log 2024-09-22T01:01:53.537 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40886.log.gz 2024-09-22T01:01:53.537 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38269.log.gz 2024-09-22T01:01:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126535.log 2024-09-22T01:01:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78614.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52699.log 2024-09-22T01:01:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126535.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122065.log 2024-09-22T01:01:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52699.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89430.log 2024-09-22T01:01:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114030.log 2024-09-22T01:01:53.538 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74745.log 2024-09-22T01:01:53.538 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46698.log 2024-09-22T01:01:53.538 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.74745.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54958.log 2024-09-22T01:01:53.538 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46698.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30100.log 2024-09-22T01:01:53.538 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54958.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124956.log 2024-09-22T01:01:53.538 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.30100.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30100.log.gz 2024-09-22T01:01:53.538 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101864.log 2024-09-22T01:01:53.538 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124956.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30228.log 2024-09-22T01:01:53.538 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101864.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45222.log 2024-09-22T01:01:53.538 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.30228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93269.log 2024-09-22T01:01:53.538 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81050.log 2024-09-22T01:01:53.538 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93269.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51064.log 2024-09-22T01:01:53.538 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81050.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120481.log 2024-09-22T01:01:53.538 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51064.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76819.log 2024-09-22T01:01:53.539 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120481.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45771.log 2024-09-22T01:01:53.539 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66757.log 2024-09-22T01:01:53.539 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45771.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52263.log 2024-09-22T01:01:53.539 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66757.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64297.log 2024-09-22T01:01:53.539 INFO:teuthology.orchestra.run.smithi060.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.140691.log.gz 2024-09-22T01:01:53.539 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52263.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93004.log 2024-09-22T01:01:53.539 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64297.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71833.log 2024-09-22T01:01:53.539 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77670.log 2024-09-22T01:01:53.539 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102537.log 2024-09-22T01:01:53.539 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77670.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.118624.log.gz 2024-09-22T01:01:53.539 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58136.log 2024-09-22T01:01:53.539 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68808.log 2024-09-22T01:01:53.539 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109188.log 2024-09-22T01:01:53.539 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68808.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80482.log 2024-09-22T01:01:53.539 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44256.log 2024-09-22T01:01:53.539 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135000.log 2024-09-22T01:01:53.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99510.log 2024-09-22T01:01:53.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135000.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59143.log 2024-09-22T01:01:53.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70784.log 2024-09-22T01:01:53.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136935.log 2024-09-22T01:01:53.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70784.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95943.log 2024-09-22T01:01:53.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136935.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35837.log 2024-09-22T01:01:53.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.95943.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127163.log 2024-09-22T01:01:53.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35837.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61191.log 2024-09-22T01:01:53.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72227.log 2024-09-22T01:01:53.540 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96365.log.gz 2024-09-22T01:01:53.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61191.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.61191.log.gz 2024-09-22T01:01:53.540 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90479.log 2024-09-22T01:01:53.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.72227.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83512.log 2024-09-22T01:01:53.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85698.log 2024-09-22T01:01:53.541 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130528.log 2024-09-22T01:01:53.541 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.85698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135966.log 2024-09-22T01:01:53.541 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100542.log.gz 2024-09-22T01:01:53.541 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.130528.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61900.log 2024-09-22T01:01:53.541 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135966.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.47538.log.gz 2024-09-22T01:01:53.541 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129667.log 2024-09-22T01:01:53.541 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46530.log.gz 2024-09-22T01:01:53.541 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61900.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67213.log 2024-09-22T01:01:53.541 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33358.log 2024-09-22T01:01:53.541 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86066.log 2024-09-22T01:01:53.541 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33358.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46062.log 2024-09-22T01:01:53.541 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.86066.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61202.log 2024-09-22T01:01:53.541 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82625.log 2024-09-22T01:01:53.541 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61202.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141383.log 2024-09-22T01:01:53.542 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82625.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29366.log 2024-09-22T01:01:53.542 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.141383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83740.log 2024-09-22T01:01:53.542 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.29366.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.141383.log.gz 2024-09-22T01:01:53.542 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29366.log.gz 2024-09-22T01:01:53.542 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83431.log 2024-09-22T01:01:53.542 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76159.log 2024-09-22T01:01:53.542 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116876.log 2024-09-22T01:01:53.542 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57044.log.gz 2024-09-22T01:01:53.542 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106726.log 2024-09-22T01:01:53.542 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116876.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70302.log 2024-09-22T01:01:53.542 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106726.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48713.log 2024-09-22T01:01:53.542 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70302.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94131.log 2024-09-22T01:01:53.542 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48713.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113964.log 2024-09-22T01:01:53.542 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132492.log 2024-09-22T01:01:53.543 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113964.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.117106.log.gz 2024-09-22T01:01:53.543 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100057.log 2024-09-22T01:01:53.543 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.117455.log.gz 2024-09-22T01:01:53.543 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.132492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96151.log 2024-09-22T01:01:53.543 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100057.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141301.log 2024-09-22T01:01:53.543 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.96151.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95097.log 2024-09-22T01:01:53.543 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.141301.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105265.log 2024-09-22T01:01:53.543 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141301.log.gz 2024-09-22T01:01:53.543 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.95097.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101318.log 2024-09-22T01:01:53.543 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105265.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33488.log 2024-09-22T01:01:53.543 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101318.log: gzip 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57967.log.gz 2024-09-22T01:01:53.543 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.63182.log 2024-09-22T01:01:53.543 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33488.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138975.log 2024-09-22T01:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.63182.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63182.log.gz 2024-09-22T01:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39801.log 2024-09-22T01:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138975.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53015.log 2024-09-22T01:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39801.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59774.log 2024-09-22T01:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.53015.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113820.log 2024-09-22T01:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126661.log 2024-09-22T01:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53598.log 2024-09-22T01:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126661.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135378.log 2024-09-22T01:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53598.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82187.log 2024-09-22T01:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135378.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128214.log 2024-09-22T01:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82187.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108635.log.gz 2024-09-22T01:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59144.log 2024-09-22T01:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128214.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128214.log.gz 2024-09-22T01:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122153.log 2024-09-22T01:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59144.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138668.log 2024-09-22T01:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122153.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139219.log 2024-09-22T01:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138668.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136808.log 2024-09-22T01:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.139219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36552.log 2024-09-22T01:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136808.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74090.log 2024-09-22T01:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103419.log 2024-09-22T01:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.74090.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123932.log 2024-09-22T01:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103419.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86582.log 2024-09-22T01:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55911.log.gz 2024-09-22T01:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123932.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94108.log 2024-09-22T01:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.86582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58765.log 2024-09-22T01:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94108.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98823.log 2024-09-22T01:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58765.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116506.log 2024-09-22T01:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98823.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76039.log 2024-09-22T01:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116506.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110037.log 2024-09-22T01:01:53.546 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70698.log 2024-09-22T01:01:53.546 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.110037.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96284.log 2024-09-22T01:01:53.546 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111677.log 2024-09-22T01:01:53.546 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96284.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97034.log 2024-09-22T01:01:53.546 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111677.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95078.log 2024-09-22T01:01:53.546 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.97034.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33129.log 2024-09-22T01:01:53.546 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.95078.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46529.log 2024-09-22T01:01:53.546 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33129.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93309.log 2024-09-22T01:01:53.546 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46529.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126282.log 2024-09-22T01:01:53.546 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54768.log 2024-09-22T01:01:53.546 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126282.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57254.log 2024-09-22T01:01:53.546 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54768.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43260.log 2024-09-22T01:01:53.546 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.50817.log.gz 2024-09-22T01:01:53.546 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57254.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123132.log 2024-09-22T01:01:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119559.log 2024-09-22T01:01:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137187.log 2024-09-22T01:01:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.119559.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121599.log 2024-09-22T01:01:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137187.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118865.log 2024-09-22T01:01:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121599.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36341.log 2024-09-22T01:01:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118865.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94545.log 2024-09-22T01:01:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68934.log 2024-09-22T01:01:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130320.log 2024-09-22T01:01:53.547 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80902.log.gz 2024-09-22T01:01:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68934.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81280.log 2024-09-22T01:01:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.130320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136217.log 2024-09-22T01:01:53.590 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81280.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97604.log 2024-09-22T01:01:53.590 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62319.log 2024-09-22T01:01:53.590 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.97604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138493.log 2024-09-22T01:01:53.590 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.62319.log: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.80987.log.gz 2024-09-22T01:01:53.590 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42460.log 2024-09-22T01:01:53.591 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31285.log 2024-09-22T01:01:53.591 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102283.log 2024-09-22T01:01:53.591 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31285.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43454.log 2024-09-22T01:01:53.591 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126743.log 2024-09-22T01:01:53.591 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43454.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142368.log 2024-09-22T01:01:53.591 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126743.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107239.log 2024-09-22T01:01:53.591 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.142368.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120924.log 2024-09-22T01:01:53.591 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66943.log 2024-09-22T01:01:53.591 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120924.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30057.log 2024-09-22T01:01:53.591 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66943.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30006.log 2024-09-22T01:01:53.591 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69883.log 2024-09-22T01:01:53.591 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30006.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51596.log 2024-09-22T01:01:53.592 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47791.log 2024-09-22T01:01:53.592 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51596.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106119.log 2024-09-22T01:01:53.592 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56035.log 2024-09-22T01:01:53.592 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106119.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93964.log 2024-09-22T01:01:53.592 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56035.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60804.log 2024-09-22T01:01:53.592 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93964.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138912.log 2024-09-22T01:01:53.592 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.142368.log.gz 2024-09-22T01:01:53.592 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60804.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68137.log 2024-09-22T01:01:53.592 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121260.log 2024-09-22T01:01:53.592 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68137.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132155.log 2024-09-22T01:01:53.592 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.121260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92134.log 2024-09-22T01:01:53.592 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79473.log.gz 2024-09-22T01:01:53.592 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132155.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55951.log 2024-09-22T01:01:53.592 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92134.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42902.log 2024-09-22T01:01:53.593 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55951.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130689.log 2024-09-22T01:01:53.593 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42902.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85034.log 2024-09-22T01:01:53.593 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.130689.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138323.log 2024-09-22T01:01:53.593 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.85034.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66519.log 2024-09-22T01:01:53.593 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136978.log 2024-09-22T01:01:53.593 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31925.log 2024-09-22T01:01:53.593 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.94036.log.gz 2024-09-22T01:01:53.593 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48882.log 2024-09-22T01:01:53.593 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.94131.log.gz 2024-09-22T01:01:53.593 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31925.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140172.log 2024-09-22T01:01:53.593 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59660.log 2024-09-22T01:01:53.593 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140172.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109168.log 2024-09-22T01:01:53.593 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101526.log.gz 2024-09-22T01:01:53.594 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59660.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58325.log 2024-09-22T01:01:53.594 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109168.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29666.log 2024-09-22T01:01:53.594 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104048.log 2024-09-22T01:01:53.594 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29666.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29666.log.gz 2024-09-22T01:01:53.594 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137482.log 2024-09-22T01:01:53.594 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104048.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100351.log.gz 2024-09-22T01:01:53.594 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63633.log 2024-09-22T01:01:53.594 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44087.log 2024-09-22T01:01:53.594 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.63633.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115014.log 2024-09-22T01:01:53.594 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44087.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35399.log 2024-09-22T01:01:53.594 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115014.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76818.log 2024-09-22T01:01:53.594 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104594.log 2024-09-22T01:01:53.594 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76818.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41270.log 2024-09-22T01:01:53.594 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137567.log.gz 2024-09-22T01:01:53.594 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104594.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32498.log 2024-09-22T01:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56667.log 2024-09-22T01:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54024.log 2024-09-22T01:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47118.log 2024-09-22T01:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54024.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129706.log 2024-09-22T01:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47118.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113964.log.gz 2024-09-22T01:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76186.log 2024-09-22T01:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.129706.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46951.log 2024-09-22T01:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114449.log.gz 2024-09-22T01:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76186.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113881.log 2024-09-22T01:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46951.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45307.log 2024-09-22T01:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68514.log 2024-09-22T01:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48756.log 2024-09-22T01:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.41518.log.gz 2024-09-22T01:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139231.log 2024-09-22T01:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41726.log.gz 2024-09-22T01:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112351.log 2024-09-22T01:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139231.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.139231.log.gz 2024-09-22T01:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120062.log 2024-09-22T01:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103061.log 2024-09-22T01:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.120062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69313.log 2024-09-22T01:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65195.log 2024-09-22T01:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42136.log.gz 2024-09-22T01:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113501.log 2024-09-22T01:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65195.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129291.log 2024-09-22T01:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113501.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142348.log 2024-09-22T01:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129291.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85724.log 2024-09-22T01:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.142348.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85785.log 2024-09-22T01:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.142348.log.gz 2024-09-22T01:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.85724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125272.log 2024-09-22T01:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.85785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35797.log 2024-09-22T01:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92167.log 2024-09-22T01:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35797.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85380.log 2024-09-22T01:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.74090.log.gz 2024-09-22T01:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92167.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134497.log 2024-09-22T01:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.85380.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85380.log.gz 2024-09-22T01:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62697.log 2024-09-22T01:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134497.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66961.log 2024-09-22T01:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.62697.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125905.log 2024-09-22T01:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66961.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106672.log 2024-09-22T01:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80985.log 2024-09-22T01:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106672.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111221.log 2024-09-22T01:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.30504.log.gz 2024-09-22T01:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127792.log 2024-09-22T01:01:53.598 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.41203.log.gz 2024-09-22T01:01:53.598 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111221.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.39138.log.gz 2024-09-22T01:01:53.598 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129778.log 2024-09-22T01:01:53.598 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87132.log 2024-09-22T01:01:53.598 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129778.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125379.log 2024-09-22T01:01:53.598 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112096.log 2024-09-22T01:01:53.598 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125379.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41705.log 2024-09-22T01:01:53.598 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130347.log 2024-09-22T01:01:53.598 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40106.log.gz 2024-09-22T01:01:53.598 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41705.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31096.log 2024-09-22T01:01:53.598 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.130347.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76701.log 2024-09-22T01:01:53.598 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.31096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93142.log 2024-09-22T01:01:53.598 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76701.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58178.log 2024-09-22T01:01:53.599 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93142.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79684.log 2024-09-22T01:01:53.599 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58178.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135042.log 2024-09-22T01:01:53.599 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79684.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66757.log.gz 2024-09-22T01:01:53.599 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116846.log 2024-09-22T01:01:53.599 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135042.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.83512.log.gz 2024-09-22T01:01:53.599 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141839.log 2024-09-22T01:01:53.599 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.116846.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.116846.log.gz 2024-09-22T01:01:53.599 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123323.log 2024-09-22T01:01:53.599 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.141839.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.141839.log.gz 2024-09-22T01:01:53.599 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127266.log 2024-09-22T01:01:53.599 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43839.log 2024-09-22T01:01:53.599 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127266.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118691.log 2024-09-22T01:01:53.599 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115395.log.gz 2024-09-22T01:01:53.599 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43839.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37309.log 2024-09-22T01:01:53.599 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.118691.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90487.log 2024-09-22T01:01:53.600 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113357.log 2024-09-22T01:01:53.600 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124222.log 2024-09-22T01:01:53.600 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113357.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91882.log.gz 2024-09-22T01:01:53.600 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86813.log 2024-09-22T01:01:53.600 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65220.log 2024-09-22T01:01:53.600 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.86813.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107889.log 2024-09-22T01:01:53.600 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37836.log 2024-09-22T01:01:53.600 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107889.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.107889.log.gz 2024-09-22T01:01:53.600 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106914.log 2024-09-22T01:01:53.600 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37836.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136217.log.gz 2024-09-22T01:01:53.600 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55054.log 2024-09-22T01:01:53.600 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.106914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51776.log 2024-09-22T01:01:53.600 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55054.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102746.log 2024-09-22T01:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51776.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99267.log 2024-09-22T01:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89009.log 2024-09-22T01:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99267.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90690.log 2024-09-22T01:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127584.log 2024-09-22T01:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.130689.log.gz 2024-09-22T01:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90690.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43050.log 2024-09-22T01:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127584.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124560.log 2024-09-22T01:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43050.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57968.log 2024-09-22T01:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124560.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140910.log 2024-09-22T01:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57968.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142739.log 2024-09-22T01:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140910.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68136.log 2024-09-22T01:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.142739.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142739.log.gz 2024-09-22T01:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72018.log 2024-09-22T01:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135842.log 2024-09-22T01:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.72018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121193.log 2024-09-22T01:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.137354.log.gz 2024-09-22T01:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97806.log 2024-09-22T01:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69104.log 2024-09-22T01:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.97806.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119404.log 2024-09-22T01:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69104.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61043.log 2024-09-22T01:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.119404.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73964.log 2024-09-22T01:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61043.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.61043.log.gz 2024-09-22T01:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86040.log 2024-09-22T01:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.73964.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77917.log 2024-09-22T01:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.86040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92999.log 2024-09-22T01:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77917.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134159.log 2024-09-22T01:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92999.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119310.log 2024-09-22T01:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134159.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97490.log 2024-09-22T01:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.119310.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110290.log 2024-09-22T01:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.97490.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36890.log 2024-09-22T01:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110290.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95128.log 2024-09-22T01:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36890.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80187.log 2024-09-22T01:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.95128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47201.log 2024-09-22T01:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105057.log 2024-09-22T01:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36849.log 2024-09-22T01:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105057.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73753.log 2024-09-22T01:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36849.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82835.log 2024-09-22T01:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73753.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52066.log 2024-09-22T01:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82835.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49680.log 2024-09-22T01:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.52066.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52066.log.gz 2024-09-22T01:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53366.log.gz 2024-09-22T01:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75175.log 2024-09-22T01:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49680.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54998.log 2024-09-22T01:01:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.75175.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123172.log 2024-09-22T01:01:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54998.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121365.log 2024-09-22T01:01:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123172.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70656.log 2024-09-22T01:01:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75885.log 2024-09-22T01:01:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70656.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67152.log 2024-09-22T01:01:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75885.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87509.log 2024-09-22T01:01:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116296.log 2024-09-22T01:01:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87509.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.70236.log.gz 2024-09-22T01:01:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56565.log 2024-09-22T01:01:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116296.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93268.log 2024-09-22T01:01:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56565.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86814.log 2024-09-22T01:01:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93268.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29441.log 2024-09-22T01:01:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.86814.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40149.log 2024-09-22T01:01:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29441.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29441.log.gz 2024-09-22T01:01:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32414.log 2024-09-22T01:01:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40149.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135378.log.gz 2024-09-22T01:01:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83204.log 2024-09-22T01:01:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32414.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90789.log 2024-09-22T01:01:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83204.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.83204.log.gz 2024-09-22T01:01:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40749.log 2024-09-22T01:01:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90789.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81868.log 2024-09-22T01:01:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40749.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40749.log.gz 2024-09-22T01:01:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36971.log 2024-09-22T01:01:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64843.log 2024-09-22T01:01:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31731.log 2024-09-22T01:01:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64843.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122065.log.gz 2024-09-22T01:01:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135883.log 2024-09-22T01:01:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31731.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61953.log 2024-09-22T01:01:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103693.log 2024-09-22T01:01:53.623 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61953.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136344.log 2024-09-22T01:01:53.623 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103693.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107119.log 2024-09-22T01:01:53.623 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.75932.log.gz 2024-09-22T01:01:53.623 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.75885.log.gz 2024-09-22T01:01:53.623 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136344.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130063.log 2024-09-22T01:01:53.623 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.107119.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81825.log 2024-09-22T01:01:53.623 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.130063.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65239.log 2024-09-22T01:01:53.623 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81825.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85035.log 2024-09-22T01:01:53.623 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65239.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107242.log 2024-09-22T01:01:53.624 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.85035.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65498.log 2024-09-22T01:01:53.624 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107242.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52118.log 2024-09-22T01:01:53.624 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131259.log 2024-09-22T01:01:53.624 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.52118.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52118.log.gz 2024-09-22T01:01:53.624 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104531.log 2024-09-22T01:01:53.624 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.131259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79851.log 2024-09-22T01:01:53.624 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104531.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99332.log 2024-09-22T01:01:53.624 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132304.log 2024-09-22T01:01:53.624 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114836.log 2024-09-22T01:01:53.624 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132304.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142120.log 2024-09-22T01:01:53.624 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114836.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110291.log 2024-09-22T01:01:53.624 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114971.log.gz 2024-09-22T01:01:53.624 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.142120.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125964.log 2024-09-22T01:01:53.625 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.142120.log.gz 2024-09-22T01:01:53.625 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110291.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115014.log.gz 2024-09-22T01:01:53.625 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98638.log 2024-09-22T01:01:53.625 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125964.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141261.log 2024-09-22T01:01:53.625 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98638.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59522.log 2024-09-22T01:01:53.625 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85838.log.gz 2024-09-22T01:01:53.625 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.141261.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.85785.log.gz 2024-09-22T01:01:53.625 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141261.log.gz 2024-09-22T01:01:53.625 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87068.log 2024-09-22T01:01:53.625 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59522.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81447.log 2024-09-22T01:01:53.625 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43678.log 2024-09-22T01:01:53.625 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44761.log 2024-09-22T01:01:53.625 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43678.log: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.30866.log.gz 2024-09-22T01:01:53.625 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126006.log 2024-09-22T01:01:53.626 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44761.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44593.log 2024-09-22T01:01:53.626 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126006.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48252.log 2024-09-22T01:01:53.626 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44593.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38914.log 2024-09-22T01:01:53.626 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48252.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37986.log 2024-09-22T01:01:53.626 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8%/var/log/ceph/cluster1-client.admin.38914.log: -- replaced with /var/log/ceph/cluster1-client.admin.31390.log.gz 2024-09-22T01:01:53.626 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76753.log 2024-09-22T01:01:53.626 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.37986.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66623.log 2024-09-22T01:01:53.626 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94534.log 2024-09-22T01:01:53.626 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66623.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79263.log 2024-09-22T01:01:53.626 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.94534.log: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43050.log.gz 2024-09-22T01:01:53.626 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94534.log.gz 2024-09-22T01:01:53.626 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138955.log 2024-09-22T01:01:53.626 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79263.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42289.log 2024-09-22T01:01:53.626 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138955.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34872.log 2024-09-22T01:01:53.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.42289.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73564.log 2024-09-22T01:01:53.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34872.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93770.log 2024-09-22T01:01:53.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73564.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63499.log 2024-09-22T01:01:53.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95572.log 2024-09-22T01:01:53.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.63499.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77876.log 2024-09-22T01:01:53.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.95572.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57213.log 2024-09-22T01:01:53.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77876.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35562.log 2024-09-22T01:01:53.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40725.log 2024-09-22T01:01:53.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114616.log 2024-09-22T01:01:53.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40725.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40725.log.gz 2024-09-22T01:01:53.627 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106941.log 2024-09-22T01:01:53.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134077.log 2024-09-22T01:01:53.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.106941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135338.log 2024-09-22T01:01:53.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134077.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141866.log 2024-09-22T01:01:53.628 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135338.log: 68.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130528.log.gz 2024-09-22T01:01:53.628 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129706.log.gz 2024-09-22T01:01:53.628 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39129.log 2024-09-22T01:01:53.628 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.141866.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110828.log 2024-09-22T01:01:53.628 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39129.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106943.log 2024-09-22T01:01:53.628 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110828.log: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.141866.log.gz 2024-09-22T01:01:53.628 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.140119.log 2024-09-22T01:01:53.628 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.106943.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106905.log 2024-09-22T01:01:53.628 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.140119.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142140.log 2024-09-22T01:01:53.628 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.106905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85899.log 2024-09-22T01:01:53.628 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.142140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83552.log 2024-09-22T01:01:53.628 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.85899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29921.log 2024-09-22T01:01:53.628 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58093.log 2024-09-22T01:01:53.629 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.29921.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117878.log 2024-09-22T01:01:53.629 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59651.log 2024-09-22T01:01:53.629 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.117878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28979.log 2024-09-22T01:01:53.629 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77144.log 2024-09-22T01:01:53.629 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51080.log.gz 2024-09-22T01:01:53.629 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51104.log.gz 2024-09-22T01:01:53.629 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.28979.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28979.log.gz 2024-09-22T01:01:53.629 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105309.log 2024-09-22T01:01:53.629 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77144.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90380.log 2024-09-22T01:01:53.629 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91043.log 2024-09-22T01:01:53.629 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90380.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.51776.log.gz 2024-09-22T01:01:53.629 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.142140.log.gz 2024-09-22T01:01:53.629 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47686.log 2024-09-22T01:01:53.629 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50206.log 2024-09-22T01:01:53.641 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59311.log 2024-09-22T01:01:53.641 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50206.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50205.log 2024-09-22T01:01:53.641 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59311.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41954.log 2024-09-22T01:01:53.641 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58325.log.gz 2024-09-22T01:01:53.641 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50205.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58051.log 2024-09-22T01:01:53.641 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131687.log 2024-09-22T01:01:53.641 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58051.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74444.log 2024-09-22T01:01:53.641 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56563.log 2024-09-22T01:01:53.641 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.74444.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113924.log 2024-09-22T01:01:53.641 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56563.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58556.log 2024-09-22T01:01:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113924.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84498.log 2024-09-22T01:01:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58556.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98148.log 2024-09-22T01:01:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84498.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.99039.log.gz 2024-09-22T01:01:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32371.log 2024-09-22T01:01:53.642 INFO:teuthology.orchestra.run.smithi060.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.98971.log.gz 2024-09-22T01:01:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98148.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43364.log 2024-09-22T01:01:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32371.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134749.log 2024-09-22T01:01:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43364.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110655.log 2024-09-22T01:01:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134749.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114728.log 2024-09-22T01:01:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110655.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30194.log 2024-09-22T01:01:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114728.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131399.log 2024-09-22T01:01:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.30194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98785.log 2024-09-22T01:01:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.131399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80211.log 2024-09-22T01:01:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109313.log 2024-09-22T01:01:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80211.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125821.log 2024-09-22T01:01:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128228.log 2024-09-22T01:01:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139173.log 2024-09-22T01:01:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.128228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34958.log 2024-09-22T01:01:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.139173.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113711.log 2024-09-22T01:01:53.643 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48756.log.gz 2024-09-22T01:01:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34958.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98112.log 2024-09-22T01:01:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113711.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51672.log 2024-09-22T01:01:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89529.log 2024-09-22T01:01:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51672.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33418.log 2024-09-22T01:01:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89529.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29062.log 2024-09-22T01:01:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33418.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58096.log 2024-09-22T01:01:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29062.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29062.log.gz 2024-09-22T01:01:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77142.log 2024-09-22T01:01:53.644 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58096.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.107243.log.gz 2024-09-22T01:01:53.644 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41020.log 2024-09-22T01:01:53.644 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77142.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103670.log 2024-09-22T01:01:53.644 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41020.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90623.log 2024-09-22T01:01:53.644 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103670.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60111.log 2024-09-22T01:01:53.644 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90623.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130308.log 2024-09-22T01:01:53.644 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60111.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125169.log 2024-09-22T01:01:53.644 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.130308.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130308.log.gz 2024-09-22T01:01:53.644 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29416.log 2024-09-22T01:01:53.644 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125169.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124224.log 2024-09-22T01:01:53.644 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.29416.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29416.log.gz 2024-09-22T01:01:53.644 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39154.log 2024-09-22T01:01:53.644 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124224.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59562.log 2024-09-22T01:01:53.644 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39154.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99081.log 2024-09-22T01:01:53.650 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59562.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129242.log 2024-09-22T01:01:53.650 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47872.log 2024-09-22T01:01:53.650 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129242.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134287.log 2024-09-22T01:01:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47872.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54768.log.gz 2024-09-22T01:01:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80209.log 2024-09-22T01:01:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134287.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122066.log 2024-09-22T01:01:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80209.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86877.log 2024-09-22T01:01:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122066.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116022.log 2024-09-22T01:01:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.86877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88110.log 2024-09-22T01:01:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116022.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138072.log 2024-09-22T01:01:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124645.log 2024-09-22T01:01:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136638.log 2024-09-22T01:01:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131880.log 2024-09-22T01:01:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136638.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82076.log 2024-09-22T01:01:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135168.log 2024-09-22T01:01:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82076.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88716.log 2024-09-22T01:01:53.652 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53387.log 2024-09-22T01:01:53.652 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43006.log 2024-09-22T01:01:53.652 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.53387.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44045.log 2024-09-22T01:01:53.652 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.43006.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45433.log 2024-09-22T01:01:53.652 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119445.log 2024-09-22T01:01:53.652 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45433.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130373.log 2024-09-22T01:01:53.652 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.119445.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71032.log 2024-09-22T01:01:53.655 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.130373.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38695.log 2024-09-22T01:01:53.655 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71032.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85557.log 2024-09-22T01:01:53.655 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38695.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55952.log 2024-09-22T01:01:53.655 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.85557.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65518.log 2024-09-22T01:01:53.655 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55952.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115507.log 2024-09-22T01:01:53.655 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65518.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47791.log.gz 2024-09-22T01:01:53.655 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74859.log 2024-09-22T01:01:53.656 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115507.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90244.log 2024-09-22T01:01:53.656 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.74859.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74403.log 2024-09-22T01:01:53.656 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90244.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83995.log 2024-09-22T01:01:53.656 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.74403.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122437.log 2024-09-22T01:01:53.656 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83995.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83380.log 2024-09-22T01:01:53.657 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38633.log 2024-09-22T01:01:53.657 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.83380.log: 90.1%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.35797.log.gz -5 2024-09-22T01:01:53.658 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.46444.log 2024-09-22T01:01:53.658 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38633.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118954.log 2024-09-22T01:01:53.658 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46444.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45686.log 2024-09-22T01:01:53.658 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.118954.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.35837.log.gz 2024-09-22T01:01:53.658 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48336.log 2024-09-22T01:01:53.659 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55275.log 2024-09-22T01:01:53.659 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48336.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73473.log 2024-09-22T01:01:53.659 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35920.log.gz 2024-09-22T01:01:53.660 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34618.log 2024-09-22T01:01:53.660 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73473.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138073.log 2024-09-22T01:01:53.661 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34618.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87805.log 2024-09-22T01:01:53.661 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111128.log 2024-09-22T01:01:53.661 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52092.log 2024-09-22T01:01:53.662 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44928.log 2024-09-22T01:01:53.662 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65220.log.gz 2024-09-22T01:01:53.662 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103693.log.gz 2024-09-22T01:01:53.662 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.52092.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52092.log.gz 2024-09-22T01:01:53.662 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37268.log 2024-09-22T01:01:53.663 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44928.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92764.log 2024-09-22T01:01:53.663 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37268.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63815.log 2024-09-22T01:01:53.663 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92764.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42079.log 2024-09-22T01:01:53.664 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.63815.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105374.log 2024-09-22T01:01:53.664 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42079.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119139.log 2024-09-22T01:01:53.665 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46273.log 2024-09-22T01:01:53.665 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45307.log.gz 2024-09-22T01:01:53.665 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.119139.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119139.log.gz 2024-09-22T01:01:53.665 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98933.log 2024-09-22T01:01:53.666 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99575.log 2024-09-22T01:01:53.666 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98933.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86942.log 2024-09-22T01:01:53.667 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99575.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66961.log.gz 2024-09-22T01:01:53.667 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103393.log 2024-09-22T01:01:53.667 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.86942.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113712.log 2024-09-22T01:01:53.668 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103393.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71876.log 2024-09-22T01:01:53.668 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135548.log 2024-09-22T01:01:53.669 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71876.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48251.log 2024-09-22T01:01:53.669 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121514.log 2024-09-22T01:01:53.669 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.80187.log.gz 2024-09-22T01:01:53.670 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134875.log 2024-09-22T01:01:53.670 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114089.log 2024-09-22T01:01:53.670 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32009.log 2024-09-22T01:01:53.671 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46083.log 2024-09-22T01:01:53.672 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124793.log 2024-09-22T01:01:53.672 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111221.log.gz 2024-09-22T01:01:53.672 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46083.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131373.log 2024-09-22T01:01:53.672 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124793.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85406.log 2024-09-22T01:01:53.673 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.131373.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106120.log 2024-09-22T01:01:53.673 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.85406.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85406.log.gz 2024-09-22T01:01:53.674 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141347.log 2024-09-22T01:01:53.674 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98638.log.gz 2024-09-22T01:01:53.674 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106120.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62591.log 2024-09-22T01:01:53.674 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.141347.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141347.log.gz 2024-09-22T01:01:53.675 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135841.log 2024-09-22T01:01:53.675 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62591.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82962.log 2024-09-22T01:01:53.676 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135841.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31514.log 2024-09-22T01:01:53.676 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99081.log.gz 2024-09-22T01:01:53.676 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82962.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95411.log 2024-09-22T01:01:53.677 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31514.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123323.log.gz 2024-09-22T01:01:53.677 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135713.log 2024-09-22T01:01:53.677 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.95411.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116443.log 2024-09-22T01:01:53.678 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135713.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35921.log 2024-09-22T01:01:53.678 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116443.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.125272.log.gz 2024-09-22T01:01:53.679 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111399.log 2024-09-22T01:01:53.679 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35921.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82833.log 2024-09-22T01:01:53.679 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121093.log 2024-09-22T01:01:53.680 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92807.log 2024-09-22T01:01:53.680 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.121093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127647.log 2024-09-22T01:01:53.681 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92807.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48797.log 2024-09-22T01:01:53.681 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127647.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117449.log 2024-09-22T01:01:53.681 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113777.log.gz 2024-09-22T01:01:53.682 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48797.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106377.log 2024-09-22T01:01:53.682 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.117449.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40974.log 2024-09-22T01:01:53.683 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106377.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32959.log 2024-09-22T01:01:53.683 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127017.log.gz 2024-09-22T01:01:53.683 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.40974.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128072.log 2024-09-22T01:01:53.683 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32959.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105631.log 2024-09-22T01:01:53.684 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128072.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128072.log.gz 2024-09-22T01:01:53.684 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132342.log 2024-09-22T01:01:53.685 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105631.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32624.log 2024-09-22T01:01:53.685 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73964.log.gz 2024-09-22T01:01:53.685 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.132342.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97464.log 2024-09-22T01:01:53.686 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32624.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52947.log 2024-09-22T01:01:53.686 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.97464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33210.log 2024-09-22T01:01:53.687 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52947.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95677.log 2024-09-22T01:01:53.687 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35772.log 2024-09-22T01:01:53.687 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90380.log.gz 2024-09-22T01:01:53.688 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.95677.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131643.log 2024-09-22T01:01:53.688 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138157.log 2024-09-22T01:01:53.689 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.131643.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109151.log 2024-09-22T01:01:53.689 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138157.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66121.log 2024-09-22T01:01:53.690 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109151.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76650.log 2024-09-22T01:01:53.690 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66121.log: gzip -5 --verbose -- 90.0% /var/log/ceph/cluster1-client.mirror.46356.log 2024-09-22T01:01:53.691 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.133048.log.gz 2024-09-22T01:01:53.691 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76650.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101696.log 2024-09-22T01:01:53.692 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46356.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130272.log 2024-09-22T01:01:53.692 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136344.log.gz 2024-09-22T01:01:53.692 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101696.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44887.log 2024-09-22T01:01:53.692 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.130272.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130272.log.gz 2024-09-22T01:01:53.693 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95436.log 2024-09-22T01:01:53.693 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44887.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125964.log.gz 2024-09-22T01:01:53.693 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104046.log 2024-09-22T01:01:53.693 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.95436.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.125905.log.gz 2024-09-22T01:01:53.694 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44426.log 2024-09-22T01:01:53.694 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104046.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53772.log 2024-09-22T01:01:53.694 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34411.log 2024-09-22T01:01:53.695 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.53772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77241.log 2024-09-22T01:01:53.695 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114836.log.gz 2024-09-22T01:01:53.695 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34411.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130662.log 2024-09-22T01:01:53.696 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77241.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125401.log 2024-09-22T01:01:53.697 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.130662.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104676.log 2024-09-22T01:01:53.697 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125401.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.87805.log.gz 2024-09-22T01:01:53.697 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92849.log 2024-09-22T01:01:53.697 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32749.log 2024-09-22T01:01:53.698 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92849.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81323.log 2024-09-22T01:01:53.698 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32749.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46697.log 2024-09-22T01:01:53.699 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81323.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81868.log.gz 2024-09-22T01:01:53.699 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32582.log 2024-09-22T01:01:53.699 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46697.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73094.log 2024-09-22T01:01:53.700 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32917.log 2024-09-22T01:01:53.701 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.73094.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91613.log 2024-09-22T01:01:53.701 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32917.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57421.log 2024-09-22T01:01:53.702 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91613.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32919.log 2024-09-22T01:01:53.702 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57421.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105923.log 2024-09-22T01:01:53.702 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135547.log 2024-09-22T01:01:53.703 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105923.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105883.log 2024-09-22T01:01:53.703 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.93142.log.gz 2024-09-22T01:01:53.703 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135547.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125191.log 2024-09-22T01:01:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77878.log 2024-09-22T01:01:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125191.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61156.log 2024-09-22T01:01:53.705 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77878.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102200.log 2024-09-22T01:01:53.705 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98239.log 2024-09-22T01:01:53.706 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79683.log 2024-09-22T01:01:53.706 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125547.log 2024-09-22T01:01:53.707 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79683.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102997.log 2024-09-22T01:01:53.708 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125547.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28925.log 2024-09-22T01:01:53.708 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134077.log.gz 2024-09-22T01:01:53.708 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102997.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43196.log 2024-09-22T01:01:53.708 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.28925.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28925.log.gz 2024-09-22T01:01:53.708 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130735.log 2024-09-22T01:01:53.709 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43196.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92344.log 2024-09-22T01:01:53.709 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.130735.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115099.log 2024-09-22T01:01:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92344.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49093.log 2024-09-22T01:01:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115099.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78967.log 2024-09-22T01:01:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49093.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96491.log 2024-09-22T01:01:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78967.log: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98933.log.gz 2024-09-22T01:01:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92092.log 2024-09-22T01:01:53.712 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40149.log.gz 2024-09-22T01:01:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96491.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.85557.log.gz 2024-09-22T01:01:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40737.log 2024-09-22T01:01:53.713 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92092.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136091.log 2024-09-22T01:01:53.713 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40737.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122646.log 2024-09-22T01:01:53.714 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40737.log.gz 2024-09-22T01:01:53.714 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67784.log 2024-09-22T01:01:53.714 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122646.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128152.log 2024-09-22T01:01:53.715 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67784.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31687.log 2024-09-22T01:01:53.715 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.128152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55994.log 2024-09-22T01:01:53.716 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.31687.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.30318.log 2024-09-22T01:01:53.716 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55994.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85744.log 2024-09-22T01:01:53.717 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.0.30318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52605.log 2024-09-22T01:01:53.717 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.85744.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53175.log 2024-09-22T01:01:53.718 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52605.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34157.log 2024-09-22T01:01:53.718 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.53175.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116192.log 2024-09-22T01:01:53.719 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34157.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52357.log 2024-09-22T01:01:53.719 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116192.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57716.log 2024-09-22T01:01:53.719 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52357.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36110.log 2024-09-22T01:01:53.720 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122879.log 2024-09-22T01:01:53.721 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139774.log 2024-09-22T01:01:53.721 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135042.log.gz 2024-09-22T01:01:53.721 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122879.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93813.log 2024-09-22T01:01:53.722 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54510.log 2024-09-22T01:01:53.722 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93813.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87252.log 2024-09-22T01:01:53.723 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.54510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74558.log 2024-09-22T01:01:53.723 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87252.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138955.log.gz 2024-09-22T01:01:53.723 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42752.log 2024-09-22T01:01:53.723 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.74558.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86978.log 2024-09-22T01:01:53.724 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89008.log 2024-09-22T01:01:53.724 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.86978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139141.log 2024-09-22T01:01:53.726 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89008.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.30795.log.gz 2024-09-22T01:01:53.726 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68204.log 2024-09-22T01:01:53.726 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.139141.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57172.log 2024-09-22T01:01:53.726 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68204.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.103061.log.gz 2024-09-22T01:01:53.726 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116589.log 2024-09-22T01:01:53.727 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57172.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107015.log 2024-09-22T01:01:53.727 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116589.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131004.log 2024-09-22T01:01:53.727 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107015.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115710.log 2024-09-22T01:01:53.728 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.131004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56122.log 2024-09-22T01:01:53.728 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115710.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130487.log 2024-09-22T01:01:53.729 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56122.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135841.log.gz 2024-09-22T01:01:53.729 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108810.log 2024-09-22T01:01:53.730 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.130487.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108179.log 2024-09-22T01:01:53.730 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.108810.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120966.log 2024-09-22T01:01:53.731 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.108179.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114762.log 2024-09-22T01:01:53.731 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.120966.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43909.log 2024-09-22T01:01:53.731 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114762.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72508.log 2024-09-22T01:01:53.732 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118135.log 2024-09-22T01:01:53.732 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72508.log: gzip -5 --verbose 90.1% -- -- replaced with /var/log/ceph/cluster1-client.mirror.66519.log.gz /var/log/ceph/cluster1-client.mirror.67992.log 2024-09-22T01:01:53.732 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:01:53.733 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84502.log 2024-09-22T01:01:53.733 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47286.log 2024-09-22T01:01:53.733 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84502.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.76650.log.gz 2024-09-22T01:01:53.733 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95800.log 2024-09-22T01:01:53.734 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47286.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75262.log 2024-09-22T01:01:53.735 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.95800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89109.log 2024-09-22T01:01:53.735 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.75262.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49412.log 2024-09-22T01:01:53.736 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89109.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56373.log 2024-09-22T01:01:53.736 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49412.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100689.log 2024-09-22T01:01:53.737 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56373.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73751.log 2024-09-22T01:01:53.737 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100689.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86851.log 2024-09-22T01:01:53.737 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43561.log.gz 2024-09-22T01:01:53.738 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77918.log 2024-09-22T01:01:53.738 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.86851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117221.log 2024-09-22T01:01:53.738 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107753.log 2024-09-22T01:01:53.739 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.117221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75403.log 2024-09-22T01:01:53.739 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42079.log.gz 2024-09-22T01:01:53.739 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139109.log 2024-09-22T01:01:53.739 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43678.log.gz 2024-09-22T01:01:53.740 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.75403.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128135.log 2024-09-22T01:01:53.741 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.139109.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.39129.log.gz 2024-09-22T01:01:53.741 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82186.log 2024-09-22T01:01:53.741 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128135.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128135.log.gz 2024-09-22T01:01:53.741 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108000.log 2024-09-22T01:01:53.742 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82186.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67926.log 2024-09-22T01:01:53.742 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.108000.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.108000.log.gz 2024-09-22T01:01:53.742 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43909.log.gz 2024-09-22T01:01:53.742 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60152.log 2024-09-22T01:01:53.743 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67926.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109420.log 2024-09-22T01:01:53.744 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110817.log 2024-09-22T01:01:53.744 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109420.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61587.log 2024-09-22T01:01:53.745 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110817.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40220.log 2024-09-22T01:01:53.745 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61587.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105719.log 2024-09-22T01:01:53.745 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44426.log.gz 2024-09-22T01:01:53.746 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70488.log 2024-09-22T01:01:53.746 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.105719.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105719.log.gz 2024-09-22T01:01:53.746 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47582.log 2024-09-22T01:01:53.747 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70488.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35503.log 2024-09-22T01:01:53.747 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47582.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119994.log 2024-09-22T01:01:53.747 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35503.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44887.log.gz 2024-09-22T01:01:53.748 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70826.log 2024-09-22T01:01:53.748 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.119994.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63727.log 2024-09-22T01:01:53.749 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70826.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46399.log 2024-09-22T01:01:53.749 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.63727.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96588.log 2024-09-22T01:01:53.750 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.32749.log.gz 2024-09-22T01:01:53.750 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71707.log 2024-09-22T01:01:53.750 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.96588.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80775.log 2024-09-22T01:01:53.751 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71707.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119378.log 2024-09-22T01:01:53.751 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39914.log 2024-09-22T01:01:53.752 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.119378.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71571.log 2024-09-22T01:01:53.752 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39914.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87869.log 2024-09-22T01:01:53.752 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71571.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112536.log 2024-09-22T01:01:53.753 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87869.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52968.log 2024-09-22T01:01:53.753 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60783.log 2024-09-22T01:01:53.754 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52968.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93184.log 2024-09-22T01:01:53.754 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60783.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122519.log 2024-09-22T01:01:53.755 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93184.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113189.log 2024-09-22T01:01:53.755 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55575.log 2024-09-22T01:01:53.756 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113189.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89783.log 2024-09-22T01:01:53.756 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55575.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85043.log 2024-09-22T01:01:53.757 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89783.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32667.log 2024-09-22T01:01:53.757 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.45686.log.gz 2024-09-22T01:01:53.757 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.85043.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97852.log 2024-09-22T01:01:53.758 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111613.log 2024-09-22T01:01:53.758 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.97852.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.admin.72946.log.gz 2024-09-22T01:01:53.758 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76901.log 2024-09-22T01:01:53.759 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111613.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71667.log 2024-09-22T01:01:53.759 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.76901.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102432.log 2024-09-22T01:01:53.760 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71667.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130829.log 2024-09-22T01:01:53.760 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102432.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51400.log 2024-09-22T01:01:53.761 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.130829.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110734.log 2024-09-22T01:01:53.761 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49641.log 2024-09-22T01:01:53.762 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110734.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91442.log 2024-09-22T01:01:53.762 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70826.log.gz 2024-09-22T01:01:53.762 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.49641.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.49641.log.gz 2024-09-22T01:01:53.762 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46656.log 2024-09-22T01:01:53.763 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115351.log 2024-09-22T01:01:53.763 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46656.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79009.log 2024-09-22T01:01:53.764 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33779.log 2024-09-22T01:01:53.764 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59439.log 2024-09-22T01:01:53.765 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33779.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76184.log 2024-09-22T01:01:53.765 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59439.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104110.log 2024-09-22T01:01:53.765 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76184.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128289.log 2024-09-22T01:01:53.767 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61899.log 2024-09-22T01:01:53.767 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128289.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128289.log.gz 2024-09-22T01:01:53.767 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139714.log 2024-09-22T01:01:53.767 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61899.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67784.log.gz 2024-09-22T01:01:53.767 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74833.log 2024-09-22T01:01:53.768 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139714.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102747.log 2024-09-22T01:01:53.768 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.74833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52471.log 2024-09-22T01:01:53.769 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102747.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131982.log 2024-09-22T01:01:53.769 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56161.log 2024-09-22T01:01:53.770 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131982.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59102.log 2024-09-22T01:01:53.770 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56161.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65307.log 2024-09-22T01:01:53.771 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59102.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47118.log.gz 2024-09-22T01:01:53.771 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72225.log 2024-09-22T01:01:53.771 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71245.log 2024-09-22T01:01:53.772 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72225.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.72225.log.gz 2024-09-22T01:01:53.772 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71454.log 2024-09-22T01:01:53.772 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71245.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97282.log 2024-09-22T01:01:53.773 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96894.log 2024-09-22T01:01:53.773 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.97282.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115669.log 2024-09-22T01:01:53.774 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.96894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44844.log 2024-09-22T01:01:53.774 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115669.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113672.log 2024-09-22T01:01:53.775 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44844.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.87869.log.gz 2024-09-22T01:01:53.775 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128205.log 2024-09-22T01:01:53.775 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113672.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72509.log 2024-09-22T01:01:53.775 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128205.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128205.log.gz 2024-09-22T01:01:53.776 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67717.log 2024-09-22T01:01:53.776 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74672.log 2024-09-22T01:01:53.777 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124432.log 2024-09-22T01:01:53.777 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.74672.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46317.log 2024-09-22T01:01:53.778 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124432.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49753.log 2024-09-22T01:01:53.779 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46317.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132129.log 2024-09-22T01:01:53.779 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90058.log 2024-09-22T01:01:53.780 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132129.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119112.log 2024-09-22T01:01:53.780 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90058.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92092.log.gz 2024-09-22T01:01:53.780 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89824.log 2024-09-22T01:01:53.780 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.119112.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119112.log.gz 2024-09-22T01:01:53.780 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52927.log 2024-09-22T01:01:53.781 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89824.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142275.log 2024-09-22T01:01:53.782 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52927.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93563.log 2024-09-22T01:01:53.782 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.142275.log: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.142275.log.gz 2024-09-22T01:01:53.782 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78423.log 2024-09-22T01:01:53.783 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93563.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133220.log 2024-09-22T01:01:53.783 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78423.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48882.log.gz 2024-09-22T01:01:53.783 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72769.log 2024-09-22T01:01:53.783 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131855.log 2024-09-22T01:01:53.784 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.72769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75976.log 2024-09-22T01:01:53.784 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131855.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47832.log 2024-09-22T01:01:53.785 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75976.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41457.log 2024-09-22T01:01:53.786 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75289.log 2024-09-22T01:01:53.786 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41457.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124642.log 2024-09-22T01:01:53.787 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.75289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121453.log 2024-09-22T01:01:53.788 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124642.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74326.log 2024-09-22T01:01:53.788 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71772.log 2024-09-22T01:01:53.788 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.74326.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74326.log.gz 2024-09-22T01:01:53.788 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136598.log 2024-09-22T01:01:53.788 INFO:teuthology.orchestra.run.smithi031.stderr: 93.5% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-09-22T01:01:53.789 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71772.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82962.log.gz 2024-09-22T01:01:53.789 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70364.log 2024-09-22T01:01:53.789 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136598.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124602.log 2024-09-22T01:01:53.790 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63158.log 2024-09-22T01:01:53.791 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60655.log 2024-09-22T01:01:53.791 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.63158.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63158.log.gz 2024-09-22T01:01:53.791 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31055.log 2024-09-22T01:01:53.792 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60655.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.100960.log.gz 2024-09-22T01:01:53.792 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134201.log 2024-09-22T01:01:53.792 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31055.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73398.log 2024-09-22T01:01:53.792 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139595.log 2024-09-22T01:01:53.793 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.73398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81449.log 2024-09-22T01:01:53.793 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139595.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112832.log 2024-09-22T01:01:53.793 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81449.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32750.log 2024-09-22T01:01:53.794 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97323.log 2024-09-22T01:01:53.794 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32750.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125402.log 2024-09-22T01:01:53.795 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.97323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100140.log 2024-09-22T01:01:53.795 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125402.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58389.log 2024-09-22T01:01:53.796 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100874.log 2024-09-22T01:01:53.796 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47246.log 2024-09-22T01:01:53.796 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100874.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34831.log 2024-09-22T01:01:53.797 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47246.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93059.log 2024-09-22T01:01:53.798 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34831.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104110.log.gz 2024-09-22T01:01:53.798 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34578.log 2024-09-22T01:01:53.798 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93059.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91110.log 2024-09-22T01:01:53.799 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34578.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61615.log 2024-09-22T01:01:53.799 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48842.log 2024-09-22T01:01:53.800 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61615.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34997.log 2024-09-22T01:01:53.800 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137610.log 2024-09-22T01:01:53.800 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.85035.log.gz 2024-09-22T01:01:53.800 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34997.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40116.log 2024-09-22T01:01:53.801 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137610.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85630.log 2024-09-22T01:01:53.801 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95286.log 2024-09-22T01:01:53.802 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.85630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116399.log 2024-09-22T01:01:53.802 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.95286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95965.log 2024-09-22T01:01:53.803 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120243.log 2024-09-22T01:01:53.803 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.95965.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30767.log 2024-09-22T01:01:53.804 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120243.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53852.log 2024-09-22T01:01:53.804 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.49680.log.gz 2024-09-22T01:01:53.804 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30767.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56413.log 2024-09-22T01:01:53.806 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53852.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.50206.log.gz 2024-09-22T01:01:53.806 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95799.log 2024-09-22T01:01:53.806 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70512.log 2024-09-22T01:01:53.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.95799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82750.log 2024-09-22T01:01:53.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112770.log 2024-09-22T01:01:53.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82750.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48293.log 2024-09-22T01:01:53.808 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92093.log 2024-09-22T01:01:53.808 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48293.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63519.log 2024-09-22T01:01:53.808 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89639.log 2024-09-22T01:01:53.809 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.63519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78968.log 2024-09-22T01:01:53.809 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39617.log 2024-09-22T01:01:53.810 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78968.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33277.log 2024-09-22T01:01:53.810 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94811.log 2024-09-22T01:01:53.810 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.105631.log.gz 2024-09-22T01:01:53.811 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33277.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113671.log 2024-09-22T01:01:53.811 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.94811.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74605.log 2024-09-22T01:01:53.812 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113671.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120176.log 2024-09-22T01:01:53.812 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.74605.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78046.log 2024-09-22T01:01:53.812 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.120176.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.114089.log.gz 2024-09-22T01:01:53.815 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78046.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110817.log.gz 2024-09-22T01:01:53.818 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125191.log.gz 2024-09-22T01:01:53.819 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.84502.log.gz 2024-09-22T01:01:53.821 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121008.log.gz 2024-09-22T01:01:53.821 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139109.log.gz 2024-09-22T01:01:53.827 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.88716.log.gz 2024-09-22T01:01:53.829 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.115669.log.gz 2024-09-22T01:01:53.831 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115710.log.gz 2024-09-22T01:01:53.831 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.107753.log.gz 2024-09-22T01:01:53.832 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107645.log.gz 2024-09-22T01:01:53.832 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119518.log.gz 2024-09-22T01:01:53.836 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111128.log.gz 2024-09-22T01:01:53.839 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124602.log.gz 2024-09-22T01:01:53.845 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.121093.log.gz 2024-09-22T01:01:53.846 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108863.log.gz 2024-09-22T01:01:53.848 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.109313.log.gz 2024-09-22T01:01:53.853 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48842.log.gz 2024-09-22T01:01:53.857 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.95436.log.gz 2024-09-22T01:01:53.857 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.95411.log.gz 2024-09-22T01:01:53.859 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.95965.log.gz 2024-09-22T01:01:53.862 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96588.log.gz 2024-09-22T01:01:53.863 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53852.log.gz 2024-09-22T01:01:53.869 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.42710.log.gz 2024-09-22T01:01:53.871 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38287.log.gz 2024-09-22T01:01:53.881 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67717.log.gz 2024-09-22T01:01:53.883 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.47582.log.gz 2024-09-22T01:01:53.891 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47246.log.gz 2024-09-22T01:01:53.896 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93563.log.gz 2024-09-22T01:01:53.897 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.81449.log.gz 2024-09-22T01:01:53.898 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100226.log.gz 2024-09-22T01:01:53.901 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.89849.log.gz 2024-09-22T01:01:53.905 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46865.log.gz 2024-09-22T01:01:53.908 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.103039.log.gz 2024-09-22T01:01:53.910 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.82791.log.gz 2024-09-22T01:01:53.912 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.112453.log.gz 2024-09-22T01:01:53.923 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.112980.log.gz 2024-09-22T01:01:53.925 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123486.log.gz 2024-09-22T01:01:53.926 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101066.log.gz 2024-09-22T01:01:53.929 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.133410.log.gz 2024-09-22T01:01:53.932 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48252.log.gz 2024-09-22T01:01:53.933 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138743.log.gz 2024-09-22T01:01:53.935 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134582.log.gz 2024-09-22T01:01:53.938 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138407.log.gz 2024-09-22T01:01:53.939 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138533.log.gz 2024-09-22T01:01:53.939 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.130414.log.gz 2024-09-22T01:01:53.942 INFO:teuthology.orchestra.run.smithi060.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63426.log.gz 2024-09-22T01:01:53.942 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63385.log.gz 2024-09-22T01:01:53.946 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104384.log.gz 2024-09-22T01:01:53.951 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136007.log.gz 2024-09-22T01:01:53.952 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.94176.log.gz 2024-09-22T01:01:53.955 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137483.log.gz 2024-09-22T01:01:53.962 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58975.log.gz 2024-09-22T01:01:53.967 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.97893.log.gz 2024-09-22T01:01:53.968 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.76012.log.gz 2024-09-22T01:01:53.975 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.39613.log.gz 2024-09-22T01:01:53.975 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.40350.log.gz 2024-09-22T01:01:53.975 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.35039.log.gz 2024-09-22T01:01:53.990 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70026.log.gz 2024-09-22T01:01:53.999 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.83737.log.gz 2024-09-22T01:01:54.001 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.81952.log.gz 2024-09-22T01:01:54.004 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.88176.log.gz 2024-09-22T01:01:54.008 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90959.log.gz 2024-09-22T01:01:54.011 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77794.log.gz 2024-09-22T01:01:54.021 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91529.log.gz 2024-09-22T01:01:54.021 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.91672.log.gz 2024-09-22T01:01:54.033 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93963.log.gz 2024-09-22T01:01:54.033 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.88484.log.gz 2024-09-22T01:01:54.040 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92513.log.gz 2024-09-22T01:01:54.044 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113925.log.gz 2024-09-22T01:01:54.045 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114029.log.gz 2024-09-22T01:01:54.049 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100058.log.gz 2024-09-22T01:01:54.050 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113882.log.gz 2024-09-22T01:01:54.053 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115920.log.gz 2024-09-22T01:01:54.055 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.118161.log.gz 2024-09-22T01:01:54.059 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122390.log.gz 2024-09-22T01:01:54.061 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122922.log.gz 2024-09-22T01:01:54.063 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.84498.log.gz 2024-09-22T01:01:54.068 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.135294.log.gz 2024-09-22T01:01:54.069 INFO:teuthology.orchestra.run.smithi060.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.97920.log.gz 2024-09-22T01:01:54.070 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136136.log.gz 2024-09-22T01:01:54.073 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.137102.log.gz 2024-09-22T01:01:54.076 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.115100.log.gz 2024-09-22T01:01:54.081 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.132129.log.gz 2024-09-22T01:01:54.085 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.125965.log.gz 2024-09-22T01:01:54.090 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.109379.log.gz 2024-09-22T01:01:54.094 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.87172.log.gz 2024-09-22T01:01:54.101 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58431.log.gz 2024-09-22T01:01:54.104 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.129767.log.gz 2024-09-22T01:01:54.110 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.131346.log.gz 2024-09-22T01:01:54.120 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57002.log.gz 2024-09-22T01:01:54.120 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55575.log.gz 2024-09-22T01:01:54.121 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56246.log.gz 2024-09-22T01:01:54.130 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54788.log.gz 2024-09-22T01:01:54.136 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56161.log.gz 2024-09-22T01:01:54.140 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56920.log.gz 2024-09-22T01:01:54.142 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56752.log.gz 2024-09-22T01:01:54.144 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.35038.log.gz 2024-09-22T01:01:54.146 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.39712.log.gz 2024-09-22T01:01:54.148 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.39762.log.gz 2024-09-22T01:01:54.150 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48422.log.gz 2024-09-22T01:01:54.153 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56875.log.gz 2024-09-22T01:01:54.159 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56204.log.gz 2024-09-22T01:01:54.165 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57716.log.gz 2024-09-22T01:01:54.179 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.33001.log.gz 2024-09-22T01:01:54.181 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47244.log.gz 2024-09-22T01:01:54.186 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80020.log.gz 2024-09-22T01:01:54.189 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46211.log.gz 2024-09-22T01:01:54.194 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126387.log.gz 2024-09-22T01:01:54.197 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92579.log.gz 2024-09-22T01:01:54.198 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81744.log.gz 2024-09-22T01:01:54.202 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92430.log.gz 2024-09-22T01:01:54.207 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.47957.log.gz 2024-09-22T01:01:54.214 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36613.log.gz 2024-09-22T01:01:54.217 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46655.log.gz 2024-09-22T01:01:54.220 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65047.log.gz 2024-09-22T01:01:54.227 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122814.log.gz 2024-09-22T01:01:54.229 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.87871.log.gz 2024-09-22T01:01:54.234 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.102577.log.gz 2024-09-22T01:01:54.234 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123972.log.gz 2024-09-22T01:01:54.236 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125651.log.gz 2024-09-22T01:01:54.239 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.119807.log.gz 2024-09-22T01:01:54.239 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.87105.log.gz 2024-09-22T01:01:54.240 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.128915.log.gz 2024-09-22T01:01:54.240 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.130061.log.gz 2024-09-22T01:01:54.244 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.108676.log.gz 2024-09-22T01:01:54.244 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124791.log.gz 2024-09-22T01:01:54.245 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100772.log.gz 2024-09-22T01:01:54.248 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68556.log.gz 2024-09-22T01:01:54.253 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.113156.log.gz 2024-09-22T01:01:54.254 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104802.log.gz 2024-09-22T01:01:54.258 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125062.log.gz 2024-09-22T01:01:54.261 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132597.log.gz 2024-09-22T01:01:54.270 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.30927.log.gz 2024-09-22T01:01:54.271 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.31029.log.gz 2024-09-22T01:01:54.275 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.75594.log.gz 2024-09-22T01:01:54.275 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58135.log.gz 2024-09-22T01:01:54.280 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.38696.log.gz 2024-09-22T01:01:54.282 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.34662.log.gz 2024-09-22T01:01:54.289 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.81365.log.gz 2024-09-22T01:01:54.293 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.61954.log.gz 2024-09-22T01:01:54.296 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47372.log.gz 2024-09-22T01:01:54.300 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104592.log.gz 2024-09-22T01:01:54.307 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64461.log.gz 2024-09-22T01:01:54.310 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64753.log.gz 2024-09-22T01:01:54.323 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89614.log.gz 2024-09-22T01:01:54.324 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.89302.log.gz 2024-09-22T01:01:54.325 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94069.log.gz 2024-09-22T01:01:54.327 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93771.log.gz 2024-09-22T01:01:54.330 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.89949.log.gz 2024-09-22T01:01:54.333 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101382.log.gz 2024-09-22T01:01:54.335 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81659.log.gz 2024-09-22T01:01:54.339 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.107508.log.gz 2024-09-22T01:01:54.341 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105269.log.gz 2024-09-22T01:01:54.344 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101571.log.gz 2024-09-22T01:01:54.346 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111654.log.gz 2024-09-22T01:01:54.350 INFO:teuthology.orchestra.run.smithi060.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.63929.log.gz 2024-09-22T01:01:54.354 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114450.log.gz 2024-09-22T01:01:54.355 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.108924.log.gz 2024-09-22T01:01:54.357 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114929.log.gz 2024-09-22T01:01:54.359 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53961.log.gz 2024-09-22T01:01:54.363 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112494.log.gz 2024-09-22T01:01:54.368 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79010.log.gz 2024-09-22T01:01:54.373 INFO:teuthology.orchestra.run.smithi060.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.108495.log.gz 2024-09-22T01:01:54.377 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.97852.log.gz 2024-09-22T01:01:54.382 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102913.log.gz 2024-09-22T01:01:54.386 INFO:teuthology.orchestra.run.smithi060.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.131601.log.gz 2024-09-22T01:01:54.392 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.109720.log.gz 2024-09-22T01:01:54.393 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.120863.log.gz 2024-09-22T01:01:54.396 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.140908.log.gz 2024-09-22T01:01:54.399 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.120149.log.gz 2024-09-22T01:01:54.401 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.106778.log.gz 2024-09-22T01:01:54.412 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57884.log.gz 2024-09-22T01:01:54.418 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.49541.log.gz 2024-09-22T01:01:54.423 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48545.log.gz 2024-09-22T01:01:54.427 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55826.log.gz 2024-09-22T01:01:54.430 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49247.log.gz 2024-09-22T01:01:54.435 INFO:teuthology.orchestra.run.smithi060.stderr: 68.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41612.log.gz 2024-09-22T01:01:54.443 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74973.log.gz 2024-09-22T01:01:54.444 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.97075.log.gz 2024-09-22T01:01:54.447 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110229.log.gz 2024-09-22T01:01:54.447 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.110185.log.gz 2024-09-22T01:01:54.451 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65904.log.gz 2024-09-22T01:01:54.456 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.admin.62259.log.gz 2024-09-22T01:01:54.464 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.48461.log.gz 2024-09-22T01:01:54.471 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.38090.log.gz 2024-09-22T01:01:54.473 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.49847.log.gz 2024-09-22T01:01:54.473 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.38926.log.gz 2024-09-22T01:01:54.481 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.62592.log.gz 2024-09-22T01:01:54.486 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40117.log.gz 2024-09-22T01:01:54.489 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.96249.log.gz 2024-09-22T01:01:54.489 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68430.log.gz 2024-09-22T01:01:54.492 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66535.log.gz 2024-09-22T01:01:54.494 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43484.log.gz 2024-09-22T01:01:54.496 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101381.log.gz 2024-09-22T01:01:54.499 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82185.log.gz 2024-09-22T01:01:54.502 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.79220.log.gz 2024-09-22T01:01:54.505 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88791.log.gz 2024-09-22T01:01:54.507 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112139.log.gz 2024-09-22T01:01:54.511 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.101063.log.gz 2024-09-22T01:01:54.513 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113611.log.gz 2024-09-22T01:01:54.520 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115459.log.gz 2024-09-22T01:01:54.526 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58011.log.gz 2024-09-22T01:01:54.531 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115668.log.gz 2024-09-22T01:01:54.535 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133739.log.gz 2024-09-22T01:01:54.538 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125232.log.gz 2024-09-22T01:01:54.540 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101172.log.gz 2024-09-22T01:01:54.541 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58096.log.gz 2024-09-22T01:01:54.546 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103270.log.gz 2024-09-22T01:01:54.550 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103376.log.gz 2024-09-22T01:01:54.553 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65964.log.gz 2024-09-22T01:01:54.556 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47076.log.gz 2024-09-22T01:01:54.557 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80273.log.gz 2024-09-22T01:01:54.569 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.62040.log.gz 2024-09-22T01:01:54.573 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89219.log.gz 2024-09-22T01:01:54.580 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67993.log.gz 2024-09-22T01:01:54.584 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69396.log.gz 2024-09-22T01:01:54.586 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68723.log.gz 2024-09-22T01:01:54.590 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79095.log.gz 2024-09-22T01:01:54.590 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69674.log.gz 2024-09-22T01:01:54.594 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43300.log.gz 2024-09-22T01:01:54.598 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.73950.log.gz 2024-09-22T01:01:54.603 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35270.log.gz 2024-09-22T01:01:54.606 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91674.log.gz 2024-09-22T01:01:54.608 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82626.log.gz 2024-09-22T01:01:54.612 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82751.log.gz 2024-09-22T01:01:54.613 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124855.log.gz 2024-09-22T01:01:54.620 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91950.log.gz 2024-09-22T01:01:54.622 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82498.log.gz 2024-09-22T01:01:54.629 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.101946.log.gz 2024-09-22T01:01:54.633 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133608.log.gz 2024-09-22T01:01:54.637 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.108354.log.gz 2024-09-22T01:01:54.638 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111570.log.gz 2024-09-22T01:01:54.638 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111569.log.gz 2024-09-22T01:01:54.642 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.135588.log.gz 2024-09-22T01:01:54.644 INFO:teuthology.orchestra.run.smithi060.stderr: 86.7% -- replaced with /var/log/ceph/cluster2-client.admin.74092.log.gz 2024-09-22T01:01:54.644 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.119007.log.gz 2024-09-22T01:01:54.656 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.62042.log.gz 2024-09-22T01:01:54.656 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.62051.log.gz 2024-09-22T01:01:54.661 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.139597.log.gz 2024-09-22T01:01:54.671 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.88109.log.gz 2024-09-22T01:01:54.671 INFO:teuthology.orchestra.run.smithi060.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41156.log.gz 2024-09-22T01:01:54.673 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42115.log.gz 2024-09-22T01:01:54.679 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42938.log.gz 2024-09-22T01:01:54.681 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.38854.log.gz 2024-09-22T01:01:54.689 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32542.log.gz 2024-09-22T01:01:54.690 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47496.log.gz 2024-09-22T01:01:54.700 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64317.log.gz 2024-09-22T01:01:54.702 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77019.log.gz 2024-09-22T01:01:54.707 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81828.log.gz 2024-09-22T01:01:54.710 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.84279.log.gz 2024-09-22T01:01:54.714 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135083.log.gz 2024-09-22T01:01:54.718 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102618.log.gz 2024-09-22T01:01:54.722 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101573.log.gz 2024-09-22T01:01:54.725 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.105373.log.gz 2024-09-22T01:01:54.732 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.85926.log.gz 2024-09-22T01:01:54.735 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111779.log.gz 2024-09-22T01:01:54.737 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121857.log.gz 2024-09-22T01:01:54.739 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.132895.log.gz 2024-09-22T01:01:54.743 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.123509.log.gz 2024-09-22T01:01:54.745 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126742.log.gz 2024-09-22T01:01:54.749 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.104846.log.gz 2024-09-22T01:01:54.753 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.118119.log.gz 2024-09-22T01:01:54.755 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.139926.log.gz 2024-09-22T01:01:54.759 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84943.log.gz 2024-09-22T01:01:54.760 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.admin.108951.log.gz 2024-09-22T01:01:54.764 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125273.log.gz 2024-09-22T01:01:54.764 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.119465.log.gz 2024-09-22T01:01:54.765 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107728.log.gz 2024-09-22T01:01:54.765 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91253.log.gz 2024-09-22T01:01:54.771 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136597.log.gz 2024-09-22T01:01:54.773 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135503.log.gz 2024-09-22T01:01:54.777 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.63902.log.gz 2024-09-22T01:01:54.788 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76268.log.gz 2024-09-22T01:01:54.792 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48378.log.gz 2024-09-22T01:01:54.798 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48672.log.gz 2024-09-22T01:01:54.798 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.49415.log.gz 2024-09-22T01:01:54.819 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56373.log.gz 2024-09-22T01:01:54.832 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65126.log.gz 2024-09-22T01:01:54.834 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.85765.log.gz 2024-09-22T01:01:54.834 INFO:teuthology.orchestra.run.smithi060.stderr: 68.3% -- replaced with /var/log/ceph/cluster1-client.mirror.86221.log.gz 2024-09-22T01:01:54.840 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69733.log.gz 2024-09-22T01:01:54.840 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43634.log.gz 2024-09-22T01:01:54.843 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67782.log.gz 2024-09-22T01:01:54.845 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46088.log.gz 2024-09-22T01:01:54.846 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67572.log.gz 2024-09-22T01:01:54.852 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44637.log.gz 2024-09-22T01:01:54.853 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103271.log.gz 2024-09-22T01:01:54.854 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.36008.log.gz 2024-09-22T01:01:54.866 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.79514.log.gz 2024-09-22T01:01:54.868 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90159.log.gz 2024-09-22T01:01:54.875 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.103753.log.gz 2024-09-22T01:01:54.877 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121778.log.gz 2024-09-22T01:01:54.879 INFO:teuthology.orchestra.run.smithi060.stderr: 54.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30279.log.gz 2024-09-22T01:01:54.879 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100350.log.gz 2024-09-22T01:01:54.881 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135085.log.gz 2024-09-22T01:01:54.885 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110906.log.gz 2024-09-22T01:01:54.886 INFO:teuthology.orchestra.run.smithi060.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109018.log.gz 2024-09-22T01:01:54.888 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123276.log.gz 2024-09-22T01:01:54.890 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.39324.log.gz 2024-09-22T01:01:54.894 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.92723.log.gz 2024-09-22T01:01:54.897 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134288.log.gz 2024-09-22T01:01:54.898 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48965.log.gz 2024-09-22T01:01:54.905 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114869.log.gz 2024-09-22T01:01:54.905 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52377.log.gz 2024-09-22T01:01:54.907 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.108012.log.gz 2024-09-22T01:01:54.914 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59272.log.gz 2024-09-22T01:01:54.918 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83107.log.gz 2024-09-22T01:01:54.919 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83244.log.gz 2024-09-22T01:01:54.920 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.83795.log.gz 2024-09-22T01:01:54.930 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.81741.log.gz 2024-09-22T01:01:54.939 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32581.log.gz 2024-09-22T01:01:54.945 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.86619.log.gz 2024-09-22T01:01:54.947 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.39551.log.gz 2024-09-22T01:01:54.947 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53624.log.gz 2024-09-22T01:01:54.951 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45012.log.gz 2024-09-22T01:01:54.953 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40141.log.gz 2024-09-22T01:01:54.957 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.62749.log.gz 2024-09-22T01:01:54.958 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64987.log.gz 2024-09-22T01:01:54.958 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62467.log.gz 2024-09-22T01:01:54.962 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67086.log.gz 2024-09-22T01:01:54.964 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77244.log.gz 2024-09-22T01:01:54.965 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.61719.log.gz 2024-09-22T01:01:54.969 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.84026.log.gz 2024-09-22T01:01:54.970 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69882.log.gz 2024-09-22T01:01:54.972 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82458.log.gz 2024-09-22T01:01:54.973 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92512.log.gz 2024-09-22T01:01:54.977 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.92639.log.gz 2024-09-22T01:01:54.984 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65158.log.gz 2024-09-22T01:01:54.988 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104718.log.gz 2024-09-22T01:01:54.989 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.105372.log.gz 2024-09-22T01:01:54.991 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.137442.log.gz 2024-09-22T01:01:54.994 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.93353.log.gz 2024-09-22T01:01:54.995 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110589.log.gz 2024-09-22T01:01:54.997 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115247.log.gz 2024-09-22T01:01:55.001 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127377.log.gz 2024-09-22T01:01:55.001 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127437.log.gz 2024-09-22T01:01:55.005 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127226.log.gz 2024-09-22T01:01:55.010 INFO:teuthology.orchestra.run.smithi060.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.admin.108837.log.gz 2024-09-22T01:01:55.010 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103482.log.gz 2024-09-22T01:01:55.016 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.33948.log.gz 2024-09-22T01:01:55.018 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93060.log.gz 2024-09-22T01:01:55.022 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69253.log.gz 2024-09-22T01:01:55.028 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.137692.log.gz 2024-09-22T01:01:55.036 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137862.log.gz 2024-09-22T01:01:55.039 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77436.log.gz 2024-09-22T01:01:55.045 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129465.log.gz 2024-09-22T01:01:55.047 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.97718.log.gz 2024-09-22T01:01:55.047 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.118384.log.gz 2024-09-22T01:01:55.047 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.118383.log.gz 2024-09-22T01:01:55.049 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.95437.log.gz 2024-09-22T01:01:55.055 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.94503.log.gz 2024-09-22T01:01:55.056 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.130601.log.gz 2024-09-22T01:01:55.064 INFO:teuthology.orchestra.run.smithi060.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.99144.log.gz 2024-09-22T01:01:55.065 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.63788.log.gz 2024-09-22T01:01:55.066 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51866.log.gz 2024-09-22T01:01:55.075 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58684.log.gz 2024-09-22T01:01:55.075 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.53102.log.gz 2024-09-22T01:01:55.083 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53816.log.gz 2024-09-22T01:01:55.086 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.39073.log.gz 2024-09-22T01:01:55.086 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54294.log.gz 2024-09-22T01:01:55.087 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41913.log.gz 2024-09-22T01:01:55.089 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.108334.log.gz 2024-09-22T01:01:55.094 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66008.log.gz 2024-09-22T01:01:55.094 INFO:teuthology.orchestra.run.smithi031.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-09-22T01:01:55.098 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44847.log.gz 2024-09-22T01:01:55.103 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59481.log.gz 2024-09-22T01:01:55.105 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68832.log.gz 2024-09-22T01:01:55.108 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88084.log.gz 2024-09-22T01:01:55.109 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 has been restored 2024-09-22T01:01:55.111 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68303.log.gz 2024-09-22T01:01:55.113 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.50207.log.gz 2024-09-22T01:01:55.115 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.110562.log.gz 2024-09-22T01:01:55.118 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82329.log.gz 2024-09-22T01:01:55.120 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112559.log.gz 2024-09-22T01:01:55.120 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.105664.log.gz 2024-09-22T01:01:55.124 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45306.log.gz 2024-09-22T01:01:55.134 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114594.log.gz 2024-09-22T01:01:55.135 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125903.log.gz 2024-09-22T01:01:55.138 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114406.log.gz 2024-09-22T01:01:55.139 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.126112.log.gz 2024-09-22T01:01:55.140 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.128830.log.gz 2024-09-22T01:01:55.143 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73582.log.gz 2024-09-22T01:01:55.146 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133478.log.gz 2024-09-22T01:01:55.151 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.133699.log.gz 2024-09-22T01:01:55.153 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134209.log.gz 2024-09-22T01:01:55.155 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125336.log.gz 2024-09-22T01:01:55.163 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135799.log.gz 2024-09-22T01:01:55.165 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136343.log.gz 2024-09-22T01:01:55.168 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.136469.log.gz 2024-09-22T01:01:55.173 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133567.log.gz 2024-09-22T01:01:55.175 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.111396.log.gz 2024-09-22T01:01:55.180 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.131563.log.gz 2024-09-22T01:01:55.189 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.140911.log.gz 2024-09-22T01:01:55.195 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78461.log.gz 2024-09-22T01:01:55.198 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.117158.log.gz 2024-09-22T01:01:55.198 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.117449.log.gz 2024-09-22T01:01:55.198 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117035.log.gz 2024-09-22T01:01:55.200 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.118026.log.gz 2024-09-22T01:01:55.202 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.121556.log.gz 2024-09-22T01:01:55.203 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.121489.log.gz 2024-09-22T01:01:55.203 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.131212.log.gz 2024-09-22T01:01:55.219 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.47621.log.gz 2024-09-22T01:01:55.224 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56078.log.gz 2024-09-22T01:01:55.232 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57421.log.gz 2024-09-22T01:01:55.237 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.39134.log.gz 2024-09-22T01:01:55.239 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.49685.log.gz 2024-09-22T01:01:55.239 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55443.log.gz 2024-09-22T01:01:55.244 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55491.log.gz 2024-09-22T01:01:55.249 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64586.log.gz 2024-09-22T01:01:55.250 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54171.log.gz 2024-09-22T01:01:55.256 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.49143.log.gz 2024-09-22T01:01:55.256 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.48421.log.gz 2024-09-22T01:01:55.257 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.84659.log.gz 2024-09-22T01:01:55.263 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.49949.log.gz 2024-09-22T01:01:55.279 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.94915.log.gz 2024-09-22T01:01:55.280 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47663.log.gz 2024-09-22T01:01:55.291 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54874.log.gz 2024-09-22T01:01:55.295 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.97665.log.gz 2024-09-22T01:01:55.298 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.108240.log.gz 2024-09-22T01:01:55.303 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121304.log.gz 2024-09-22T01:01:55.307 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118942.log.gz 2024-09-22T01:01:55.313 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.61498.log.gz 2024-09-22T01:01:55.316 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123889.log.gz 2024-09-22T01:01:55.323 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109784.log.gz 2024-09-22T01:01:55.325 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110079.log.gz 2024-09-22T01:01:55.327 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67424.log.gz 2024-09-22T01:01:55.330 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47411.log.gz 2024-09-22T01:01:55.330 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124987.log.gz 2024-09-22T01:01:55.332 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90059.log.gz 2024-09-22T01:01:55.334 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.103794.log.gz 2024-09-22T01:01:55.335 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136428.log.gz 2024-09-22T01:01:55.336 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91108.log.gz 2024-09-22T01:01:55.338 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79389.log.gz 2024-09-22T01:01:55.342 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79135.log.gz 2024-09-22T01:01:55.342 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.98891.log.gz 2024-09-22T01:01:55.343 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107631.log.gz 2024-09-22T01:01:55.345 INFO:teuthology.orchestra.run.smithi060.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.admin.128530.log.gz 2024-09-22T01:01:55.345 INFO:teuthology.orchestra.run.smithi060.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.130642.log.gz 2024-09-22T01:01:55.345 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.104216.log.gz 2024-09-22T01:01:55.346 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124853.log.gz 2024-09-22T01:01:55.354 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.35168.log.gz 2024-09-22T01:01:55.361 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116340.log.gz 2024-09-22T01:01:55.363 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.38087.log.gz 2024-09-22T01:01:55.367 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123803.log.gz 2024-09-22T01:01:55.367 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123819.log.gz 2024-09-22T01:01:55.369 INFO:teuthology.orchestra.run.smithi060.stderr: 68.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86335.log.gz 2024-09-22T01:01:55.369 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.86241.log.gz 2024-09-22T01:01:55.370 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.76079.log.gz 2024-09-22T01:01:55.375 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36846.log.gz 2024-09-22T01:01:55.381 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58807.log.gz 2024-09-22T01:01:55.391 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84734.log.gz 2024-09-22T01:01:55.392 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.85039.log.gz 2024-09-22T01:01:55.393 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.98064.log.gz 2024-09-22T01:01:55.395 INFO:teuthology.orchestra.run.smithi060.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40927.log.gz 2024-09-22T01:01:55.396 INFO:teuthology.orchestra.run.smithi060.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41042.log.gz 2024-09-22T01:01:55.397 INFO:teuthology.orchestra.run.smithi060.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.64357.log.gz 2024-09-22T01:01:55.402 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.45729.log.gz 2024-09-22T01:01:55.405 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.33088.log.gz 2024-09-22T01:01:55.406 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67214.log.gz 2024-09-22T01:01:55.409 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46357.log.gz 2024-09-22T01:01:55.411 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69943.log.gz 2024-09-22T01:01:55.411 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65369.log.gz 2024-09-22T01:01:55.416 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43777.log.gz 2024-09-22T01:01:55.418 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92680.log.gz 2024-09-22T01:01:55.420 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100731.log.gz 2024-09-22T01:01:55.423 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103332.log.gz 2024-09-22T01:01:55.423 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102642.log.gz 2024-09-22T01:01:55.423 INFO:teuthology.orchestra.run.smithi060.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.86154.log.gz 2024-09-22T01:01:55.426 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.101801.log.gz 2024-09-22T01:01:55.427 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92368.log.gz 2024-09-22T01:01:55.429 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104742.log.gz 2024-09-22T01:01:55.432 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.93604.log.gz 2024-09-22T01:01:55.436 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99397.log.gz 2024-09-22T01:01:55.443 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125483.log.gz 2024-09-22T01:01:55.445 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137397.log.gz 2024-09-22T01:01:55.448 INFO:teuthology.orchestra.run.smithi060.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52740.log.gz 2024-09-22T01:01:55.449 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122672.log.gz 2024-09-22T01:01:55.452 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.118787.log.gz 2024-09-22T01:01:55.453 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.120202.log.gz 2024-09-22T01:01:55.453 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113124.log.gz 2024-09-22T01:01:55.453 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.119693.log.gz 2024-09-22T01:01:55.454 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132756.log.gz 2024-09-22T01:01:55.462 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.131118.log.gz 2024-09-22T01:01:55.462 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136220.log.gz 2024-09-22T01:01:55.468 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121724.log.gz 2024-09-22T01:01:55.469 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.86355.log.gz 2024-09-22T01:01:55.474 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.34747.log.gz 2024-09-22T01:01:55.476 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82330.log.gz 2024-09-22T01:01:55.479 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.91168.log.gz 2024-09-22T01:01:55.484 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135653.log.gz 2024-09-22T01:01:55.492 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.62646.log.gz 2024-09-22T01:01:55.496 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.36403.log.gz 2024-09-22T01:01:55.496 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65343.log.gz 2024-09-22T01:01:55.501 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47622.log.gz 2024-09-22T01:01:55.504 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69984.log.gz 2024-09-22T01:01:55.505 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66794.log.gz 2024-09-22T01:01:55.509 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44216.log.gz 2024-09-22T01:01:55.513 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47831.log.gz 2024-09-22T01:01:55.516 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68097.log.gz 2024-09-22T01:01:55.519 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91883.log.gz 2024-09-22T01:01:55.524 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.93860.log.gz 2024-09-22T01:01:55.526 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.92997.log.gz 2024-09-22T01:01:55.527 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.106926.log.gz 2024-09-22T01:01:55.528 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108427.log.gz 2024-09-22T01:01:55.529 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.99750.log.gz 2024-09-22T01:01:55.530 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104382.log.gz 2024-09-22T01:01:55.533 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114239.log.gz 2024-09-22T01:01:55.544 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70993.log.gz 2024-09-22T01:01:55.545 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116341.log.gz 2024-09-22T01:01:55.546 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102408.log.gz 2024-09-22T01:01:55.551 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88465.log.gz 2024-09-22T01:01:55.553 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101527.log.gz 2024-09-22T01:01:55.554 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.84785.log.gz 2024-09-22T01:01:55.555 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.133607.log.gz 2024-09-22T01:01:55.556 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.88851.log.gz 2024-09-22T01:01:55.557 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.109065.log.gz 2024-09-22T01:01:55.559 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91800.log.gz 2024-09-22T01:01:55.561 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.87317.log.gz 2024-09-22T01:01:55.563 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.130575.log.gz 2024-09-22T01:01:55.567 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112622.log.gz 2024-09-22T01:01:55.574 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56541.log.gz 2024-09-22T01:01:55.577 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93396.log.gz 2024-09-22T01:01:55.578 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57296.log.gz 2024-09-22T01:01:55.583 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.125796.log.gz 2024-09-22T01:01:55.584 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122307.log.gz 2024-09-22T01:01:55.588 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58597.log.gz 2024-09-22T01:01:55.591 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60571.log.gz 2024-09-22T01:01:55.596 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36509.log.gz 2024-09-22T01:01:55.600 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60446.log.gz 2024-09-22T01:01:55.601 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.38698.log.gz 2024-09-22T01:01:55.606 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.50710.log.gz 2024-09-22T01:01:55.608 INFO:teuthology.orchestra.run.smithi060.stderr: 80.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42333.log.gz 2024-09-22T01:01:55.613 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43048.log.gz 2024-09-22T01:01:55.617 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60826.log.gz 2024-09-22T01:01:55.620 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45559.log.gz 2024-09-22T01:01:55.628 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67087.log.gz 2024-09-22T01:01:55.636 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69252.log.gz 2024-09-22T01:01:55.645 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.135337.log.gz 2024-09-22T01:01:55.652 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68766.log.gz 2024-09-22T01:01:55.656 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81406.log.gz 2024-09-22T01:01:55.659 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.140274.log.gz 2024-09-22T01:01:55.662 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79388.log.gz 2024-09-22T01:01:55.667 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.101778.log.gz 2024-09-22T01:01:55.667 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130849.log.gz 2024-09-22T01:01:55.668 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.112516.log.gz 2024-09-22T01:01:55.675 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90999.log.gz 2024-09-22T01:01:55.678 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89850.log.gz 2024-09-22T01:01:55.678 INFO:teuthology.orchestra.run.smithi060.stderr: 68.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97987.log.gz 2024-09-22T01:01:55.681 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131419.log.gz 2024-09-22T01:01:55.681 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136302.log.gz 2024-09-22T01:01:55.682 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.90372.log.gz 2024-09-22T01:01:55.685 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125902.log.gz 2024-09-22T01:01:55.686 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114763.log.gz 2024-09-22T01:01:55.686 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.119008.log.gz 2024-09-22T01:01:55.687 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.130434.log.gz 2024-09-22T01:01:55.688 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.119237.log.gz 2024-09-22T01:01:55.688 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118260.log.gz 2024-09-22T01:01:55.689 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101947.log.gz 2024-09-22T01:01:55.698 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133992.log.gz 2024-09-22T01:01:55.706 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53433.log.gz 2024-09-22T01:01:55.713 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78255.log.gz 2024-09-22T01:01:55.714 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.75651.log.gz 2024-09-22T01:01:55.716 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33989.log.gz 2024-09-22T01:01:55.726 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.129116.log.gz 2024-09-22T01:01:55.728 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.121072.log.gz 2024-09-22T01:01:55.730 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.140916.log.gz 2024-09-22T01:01:55.735 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98407.log.gz 2024-09-22T01:01:55.737 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.129744.log.gz 2024-09-22T01:01:55.742 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109932.log.gz 2024-09-22T01:01:55.750 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57424.log.gz 2024-09-22T01:01:55.755 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.108582.log.gz 2024-09-22T01:01:55.756 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.72893.log.gz 2024-09-22T01:01:55.756 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.73161.log.gz 2024-09-22T01:01:55.758 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74517.log.gz 2024-09-22T01:01:55.761 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110265.log.gz 2024-09-22T01:01:55.762 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.110520.log.gz 2024-09-22T01:01:55.771 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54469.log.gz 2024-09-22T01:01:55.772 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49135.log.gz 2024-09-22T01:01:55.776 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49052.log.gz 2024-09-22T01:01:55.782 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54256.log.gz 2024-09-22T01:01:55.782 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.118868.log.gz 2024-09-22T01:01:55.795 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.140332.log.gz 2024-09-22T01:01:55.801 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112245.log.gz 2024-09-22T01:01:55.802 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.112306.log.gz 2024-09-22T01:01:55.805 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104426.log.gz 2024-09-22T01:01:55.808 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113819.log.gz 2024-09-22T01:01:55.811 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.124770.log.gz 2024-09-22T01:01:55.816 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.90034.log.gz 2024-09-22T01:01:55.818 INFO:teuthology.orchestra.run.smithi060.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.108106.log.gz 2024-09-22T01:01:55.818 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.115434.log.gz 2024-09-22T01:01:55.818 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.129579.log.gz 2024-09-22T01:01:55.821 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93478.log.gz 2024-09-22T01:01:55.823 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126499.log.gz 2024-09-22T01:01:55.827 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111442.log.gz 2024-09-22T01:01:55.829 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.115183.log.gz 2024-09-22T01:01:55.830 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.50328.log.gz 2024-09-22T01:01:55.836 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.103480.log.gz 2024-09-22T01:01:55.837 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103586.log.gz 2024-09-22T01:01:55.840 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45642.log.gz 2024-09-22T01:01:55.843 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.63560.log.gz 2024-09-22T01:01:55.843 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.48043.log.gz 2024-09-22T01:01:55.843 INFO:teuthology.orchestra.run.smithi060.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63996.log.gz 2024-09-22T01:01:55.848 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36551.log.gz 2024-09-22T01:01:55.853 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54590.log.gz 2024-09-22T01:01:55.856 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60403.log.gz 2024-09-22T01:01:55.865 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122105.log.gz 2024-09-22T01:01:55.868 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.admin.73290.log.gz 2024-09-22T01:01:55.873 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62491.log.gz 2024-09-22T01:01:55.876 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65837.log.gz 2024-09-22T01:01:55.876 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69102.log.gz 2024-09-22T01:01:55.879 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69186.log.gz 2024-09-22T01:01:55.881 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.47882.log.gz 2024-09-22T01:01:55.884 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54110.log.gz 2024-09-22T01:01:55.885 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66942.log.gz 2024-09-22T01:01:55.895 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.104637.log.gz 2024-09-22T01:01:55.897 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.93209.log.gz 2024-09-22T01:01:55.909 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54404.log.gz 2024-09-22T01:01:55.914 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114198.log.gz 2024-09-22T01:01:55.916 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.106262.log.gz 2024-09-22T01:01:55.917 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.106587.log.gz 2024-09-22T01:01:55.922 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.118435.log.gz 2024-09-22T01:01:55.928 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61096.log.gz 2024-09-22T01:01:55.935 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54830.log.gz 2024-09-22T01:01:55.945 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80146.log.gz 2024-09-22T01:01:55.947 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.94812.log.gz 2024-09-22T01:01:55.948 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.82119.log.gz 2024-09-22T01:01:55.954 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.128598.log.gz 2024-09-22T01:01:55.954 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.134916.log.gz 2024-09-22T01:01:55.955 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113191.log.gz 2024-09-22T01:01:55.955 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.129753.log.gz 2024-09-22T01:01:55.957 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81785.log.gz 2024-09-22T01:01:55.958 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71123.log.gz 2024-09-22T01:01:55.962 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.108609.log.gz 2024-09-22T01:01:55.964 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.admin.51253.log.gz 2024-09-22T01:01:55.964 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115644.log.gz 2024-09-22T01:01:55.965 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101697.log.gz 2024-09-22T01:01:55.969 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137944.log.gz 2024-09-22T01:01:55.972 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.105223.log.gz 2024-09-22T01:01:55.978 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127500.log.gz 2024-09-22T01:01:55.988 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.40008.log.gz 2024-09-22T01:01:55.991 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.51182.log.gz 2024-09-22T01:01:55.994 INFO:teuthology.orchestra.run.smithi060.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53196.log.gz 2024-09-22T01:01:55.995 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44341.log.gz 2024-09-22T01:01:56.001 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66120.log.gz 2024-09-22T01:01:56.004 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67464.log.gz 2024-09-22T01:01:56.006 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66834.log.gz 2024-09-22T01:01:56.011 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139296.log.gz 2024-09-22T01:01:56.019 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57086.log.gz 2024-09-22T01:01:56.020 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82371.log.gz 2024-09-22T01:01:56.025 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.138114.log.gz 2024-09-22T01:01:56.028 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80565.log.gz 2024-09-22T01:01:56.030 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86940.log.gz 2024-09-22T01:01:56.034 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115457.log.gz 2024-09-22T01:01:56.037 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122921.log.gz 2024-09-22T01:01:56.037 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.86745.log.gz 2024-09-22T01:01:56.043 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70613.log.gz 2024-09-22T01:01:56.045 INFO:teuthology.orchestra.run.smithi060.stderr: 55.3% -- replaced with /var/log/ceph/cluster1-client.mirror.29955.log.gz 2024-09-22T01:01:56.047 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.30731.log.gz 2024-09-22T01:01:56.063 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52491.log.gz 2024-09-22T01:01:56.066 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.48881.log.gz 2024-09-22T01:01:56.070 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57547.log.gz 2024-09-22T01:01:56.076 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.119787.log.gz 2024-09-22T01:01:56.077 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119746.log.gz 2024-09-22T01:01:56.077 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.130064.log.gz 2024-09-22T01:01:56.079 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57633.log.gz 2024-09-22T01:01:56.087 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56961.log.gz 2024-09-22T01:01:56.092 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.97376.log.gz 2024-09-22T01:01:56.092 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.48588.log.gz 2024-09-22T01:01:56.096 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.110352.log.gz 2024-09-22T01:01:56.098 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118319.log.gz 2024-09-22T01:01:56.099 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40906.log.gz 2024-09-22T01:01:56.099 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.52673.log.gz 2024-09-22T01:01:56.110 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58262.log.gz 2024-09-22T01:01:56.112 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61231.log.gz 2024-09-22T01:01:56.113 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.94813.log.gz 2024-09-22T01:01:56.115 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.131817.log.gz 2024-09-22T01:01:56.116 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.106114.log.gz 2024-09-22T01:01:56.117 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86294.log.gz 2024-09-22T01:01:56.121 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58474.log.gz 2024-09-22T01:01:56.121 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.51392.log.gz 2024-09-22T01:01:56.127 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42643.log.gz 2024-09-22T01:01:56.132 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111283.log.gz 2024-09-22T01:01:56.135 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.116820.log.gz 2024-09-22T01:01:56.139 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.123657.log.gz 2024-09-22T01:01:56.143 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.137271.log.gz 2024-09-22T01:01:56.146 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79809.log.gz 2024-09-22T01:01:56.149 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.100518.log.gz 2024-09-22T01:01:56.153 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.124352.log.gz 2024-09-22T01:01:56.155 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137443.log.gz 2024-09-22T01:01:56.159 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.94066.log.gz 2024-09-22T01:01:56.162 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89069.log.gz 2024-09-22T01:01:56.164 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.100856.log.gz 2024-09-22T01:01:56.169 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.103900.log.gz 2024-09-22T01:01:56.172 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112895.log.gz 2024-09-22T01:01:56.177 INFO:teuthology.orchestra.run.smithi060.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.120129.log.gz 2024-09-22T01:01:56.177 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110460.log.gz 2024-09-22T01:01:56.177 INFO:teuthology.orchestra.run.smithi060.stderr: 56.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30262.log.gz 2024-09-22T01:01:56.180 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114826.log.gz 2024-09-22T01:01:56.185 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.46742.log.gz 2024-09-22T01:01:56.187 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.109889.log.gz 2024-09-22T01:01:56.188 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68412.log.gz 2024-09-22T01:01:56.188 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.84958.log.gz 2024-09-22T01:01:56.189 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.98301.log.gz 2024-09-22T01:01:56.191 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107699.log.gz 2024-09-22T01:01:56.192 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59062.log.gz 2024-09-22T01:01:56.192 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.128896.log.gz 2024-09-22T01:01:56.195 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68767.log.gz 2024-09-22T01:01:56.200 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69774.log.gz 2024-09-22T01:01:56.203 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59227.log.gz 2024-09-22T01:01:56.206 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138449.log.gz 2024-09-22T01:01:56.208 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134831.log.gz 2024-09-22T01:01:56.212 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.135758.log.gz 2024-09-22T01:01:56.213 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60361.log.gz 2024-09-22T01:01:56.218 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.119834.log.gz 2024-09-22T01:01:56.220 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79347.log.gz 2024-09-22T01:01:56.226 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57338.log.gz 2024-09-22T01:01:56.227 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.86127.log.gz 2024-09-22T01:01:56.233 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44929.log.gz 2024-09-22T01:01:56.234 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35080.log.gz 2024-09-22T01:01:56.234 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.39135.log.gz 2024-09-22T01:01:56.236 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59271.log.gz 2024-09-22T01:01:56.241 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.42157.log.gz 2024-09-22T01:01:56.242 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.50375.log.gz 2024-09-22T01:01:56.243 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42077.log.gz 2024-09-22T01:01:56.245 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71855.log.gz 2024-09-22T01:01:56.246 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41000.log.gz 2024-09-22T01:01:56.246 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46164.log.gz 2024-09-22T01:01:56.251 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.115079.log.gz 2024-09-22T01:01:56.253 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133867.log.gz 2024-09-22T01:01:56.259 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.127056.log.gz 2024-09-22T01:01:56.263 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101988.log.gz 2024-09-22T01:01:56.267 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70278.log.gz 2024-09-22T01:01:56.270 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70314.log.gz 2024-09-22T01:01:56.271 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101652.log.gz 2024-09-22T01:01:56.273 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92933.log.gz 2024-09-22T01:01:56.279 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90790.log.gz 2024-09-22T01:01:56.281 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102494.log.gz 2024-09-22T01:01:56.283 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.85038.log.gz 2024-09-22T01:01:56.288 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.118623.log.gz 2024-09-22T01:01:56.288 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.133908.log.gz 2024-09-22T01:01:56.290 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.106264.log.gz 2024-09-22T01:01:56.290 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100100.log.gz 2024-09-22T01:01:56.292 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.130062.log.gz 2024-09-22T01:01:56.293 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111887.log.gz 2024-09-22T01:01:56.297 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117837.log.gz 2024-09-22T01:01:56.301 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45057.log.gz 2024-09-22T01:01:56.303 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32539.log.gz 2024-09-22T01:01:56.307 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75847.log 2024-09-22T01:01:56.307 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47981.log.gz 2024-09-22T01:01:56.310 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30993.log.gz 2024-09-22T01:01:56.311 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.86563.log.gz 2024-09-22T01:01:56.315 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.32792.log.gz 2024-09-22T01:01:56.320 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.40833.log.gz 2024-09-22T01:01:56.321 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40366.log 2024-09-22T01:01:56.321 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75847.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84400.log 2024-09-22T01:01:56.322 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112141.log 2024-09-22T01:01:56.322 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31647.log 2024-09-22T01:01:56.323 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112141.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55107.log 2024-09-22T01:01:56.323 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31647.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30306.log 2024-09-22T01:01:56.324 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55053.log 2024-09-22T01:01:56.325 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55107.log: /var/log/ceph/cluster2-client.mirror.30306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69524.log 2024-09-22T01:01:56.325 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55053.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95438.log 2024-09-22T01:01:56.326 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69524.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51323.log 2024-09-22T01:01:56.326 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.95438.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32152.log 2024-09-22T01:01:56.327 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126322.log 2024-09-22T01:01:56.327 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110458.log.gz 2024-09-22T01:01:56.327 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45391.log 2024-09-22T01:01:56.327 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35798.log 2024-09-22T01:01:56.328 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72873.log 2024-09-22T01:01:56.329 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35798.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107114.log 2024-09-22T01:01:56.329 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.110653.log.gz 2024-09-22T01:01:56.330 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.72873.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98322.log 2024-09-22T01:01:56.330 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.107114.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.87841.log.gz 2024-09-22T01:01:56.330 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39675.log 2024-09-22T01:01:56.331 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.98322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65981.log 2024-09-22T01:01:56.331 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39675.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94353.log 2024-09-22T01:01:56.331 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65981.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48926.log 2024-09-22T01:01:56.332 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48211.log 2024-09-22T01:01:56.332 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48926.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33568.log 2024-09-22T01:01:56.332 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44636.log.gz 2024-09-22T01:01:56.333 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48211.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138239.log 2024-09-22T01:01:56.333 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33568.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40761.log 2024-09-22T01:01:56.334 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47915.log 2024-09-22T01:01:56.334 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40761.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40761.log.gz 2024-09-22T01:01:56.334 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133824.log 2024-09-22T01:01:56.334 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45476.log.gz 2024-09-22T01:01:56.335 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47915.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118160.log 2024-09-22T01:01:56.335 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133824.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73399.log 2024-09-22T01:01:56.336 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118160.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91419.log 2024-09-22T01:01:56.336 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45517.log.gz 2024-09-22T01:01:56.336 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68346.log 2024-09-22T01:01:56.337 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91419.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36974.log 2024-09-22T01:01:56.337 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68346.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102431.log 2024-09-22T01:01:56.338 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36974.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128913.log 2024-09-22T01:01:56.338 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.32623.log.gz 2024-09-22T01:01:56.339 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115141.log 2024-09-22T01:01:56.339 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128913.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83984.log 2024-09-22T01:01:56.340 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120336.log 2024-09-22T01:01:56.340 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83984.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28899.log 2024-09-22T01:01:56.341 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.120336.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141193.log 2024-09-22T01:01:56.341 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.28899.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28899.log.gz 2024-09-22T01:01:56.341 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127585.log 2024-09-22T01:01:56.342 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.141193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83686.log 2024-09-22T01:01:56.342 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127585.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76224.log 2024-09-22T01:01:56.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41317.log 2024-09-22T01:01:56.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.76224.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113610.log 2024-09-22T01:01:56.344 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.41317.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76704.log 2024-09-22T01:01:56.344 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65799.log 2024-09-22T01:01:56.345 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76704.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44760.log 2024-09-22T01:01:56.346 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65799.log: gzip -5 --verbose -- 90.0% /var/log/ceph/cluster1-client.mirror.43778.log -- replaced with /var/log/ceph/cluster1-client.mirror.112897.log.gz 2024-09-22T01:01:56.346 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:01:56.346 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44760.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142094.log 2024-09-22T01:01:56.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43778.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79598.log 2024-09-22T01:01:56.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.142094.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138196.log 2024-09-22T01:01:56.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79598.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78884.log 2024-09-22T01:01:56.348 INFO:teuthology.orchestra.run.smithi060.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.142094.log.gz 2024-09-22T01:01:56.349 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138196.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71328.log 2024-09-22T01:01:56.349 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78884.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101170.log 2024-09-22T01:01:56.350 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.67362.log.gz 2024-09-22T01:01:56.350 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87763.log 2024-09-22T01:01:56.350 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101170.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120434.log 2024-09-22T01:01:56.351 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.87763.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57128.log 2024-09-22T01:01:56.351 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75847.log.gz 2024-09-22T01:01:56.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.120434.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45812.log 2024-09-22T01:01:56.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87277.log 2024-09-22T01:01:56.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45812.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126450.log 2024-09-22T01:01:56.353 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87277.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68054.log 2024-09-22T01:01:56.354 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126450.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120357.log 2024-09-22T01:01:56.354 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68054.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124182.log 2024-09-22T01:01:56.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120357.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30604.log 2024-09-22T01:01:56.355 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67633.log.gz 2024-09-22T01:01:56.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33780.log 2024-09-22T01:01:56.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116927.log 2024-09-22T01:01:56.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33780.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44803.log 2024-09-22T01:01:56.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.116927.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.116927.log.gz 2024-09-22T01:01:56.357 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142594.log 2024-09-22T01:01:56.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44803.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83160.log 2024-09-22T01:01:56.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.142594.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142594.log.gz 2024-09-22T01:01:56.358 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45961.log 2024-09-22T01:01:56.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.83160.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69018.log.gz 2024-09-22T01:01:56.358 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55318.log 2024-09-22T01:01:56.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45961.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113609.log 2024-09-22T01:01:56.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140532.log 2024-09-22T01:01:56.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113609.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100476.log 2024-09-22T01:01:56.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.140532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134033.log 2024-09-22T01:01:56.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100476.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73394.log 2024-09-22T01:01:56.361 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47916.log 2024-09-22T01:01:56.361 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.73394.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138932.log 2024-09-22T01:01:56.362 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47916.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94391.log 2024-09-22T01:01:56.362 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138932.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45959.log 2024-09-22T01:01:56.363 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.94391.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94391.log.gz 2024-09-22T01:01:56.363 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97966.log 2024-09-22T01:01:56.363 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45959.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87806.log 2024-09-22T01:01:56.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.97966.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135800.log 2024-09-22T01:01:56.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87806.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115394.log 2024-09-22T01:01:56.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132995.log 2024-09-22T01:01:56.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115394.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79724.log 2024-09-22T01:01:56.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132995.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107928.log 2024-09-22T01:01:56.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65263.log 2024-09-22T01:01:56.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107928.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.107928.log.gz 2024-09-22T01:01:56.366 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113401.log 2024-09-22T01:01:56.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.65263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140914.log 2024-09-22T01:01:56.368 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113401.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50949.log 2024-09-22T01:01:56.368 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140914.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56564.log 2024-09-22T01:01:56.368 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.50949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45516.log 2024-09-22T01:01:56.369 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75014.log 2024-09-22T01:01:56.369 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45516.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38907.log 2024-09-22T01:01:56.370 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75014.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60362.log 2024-09-22T01:01:56.370 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38907.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114341.log 2024-09-22T01:01:56.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97873.log 2024-09-22T01:01:56.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32834.log 2024-09-22T01:01:56.372 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.97873.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49331.log 2024-09-22T01:01:56.372 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37987.log 2024-09-22T01:01:56.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103878.log 2024-09-22T01:01:56.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.37987.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89279.log 2024-09-22T01:01:56.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73766.log 2024-09-22T01:01:56.374 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89279.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129399.log 2024-09-22T01:01:56.374 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59985.log 2024-09-22T01:01:56.375 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.129399.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107952.log 2024-09-22T01:01:56.375 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82497.log 2024-09-22T01:01:56.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107952.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.107952.log.gz 2024-09-22T01:01:56.376 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103587.log 2024-09-22T01:01:56.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82497.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58933.log 2024-09-22T01:01:56.377 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103587.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110312.log 2024-09-22T01:01:56.377 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58933.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119290.log 2024-09-22T01:01:56.378 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29566.log 2024-09-22T01:01:56.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.119290.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91840.log 2024-09-22T01:01:56.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.29566.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29566.log.gz 2024-09-22T01:01:56.379 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80649.log 2024-09-22T01:01:56.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91840.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56288.log 2024-09-22T01:01:56.380 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80649.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47117.log 2024-09-22T01:01:56.380 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.93628.log.gz 2024-09-22T01:01:56.380 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65702.log 2024-09-22T01:01:56.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47117.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33882.log 2024-09-22T01:01:56.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.65702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82983.log 2024-09-22T01:01:56.382 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136934.log 2024-09-22T01:01:56.382 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82983.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90664.log 2024-09-22T01:01:56.383 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136934.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40361.log 2024-09-22T01:01:56.383 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90664.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59711.log 2024-09-22T01:01:56.384 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40361.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137152.log 2024-09-22T01:01:56.384 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59711.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72135.log 2024-09-22T01:01:56.385 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137152.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106775.log 2024-09-22T01:01:56.385 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72135.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.72135.log.gz 2024-09-22T01:01:56.385 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51907.log 2024-09-22T01:01:56.385 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104677.log.gz 2024-09-22T01:01:56.386 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.106775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52068.log 2024-09-22T01:01:56.386 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51907.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70426.log 2024-09-22T01:01:56.386 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.52068.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52068.log.gz 2024-09-22T01:01:56.387 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80272.log 2024-09-22T01:01:56.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138532.log 2024-09-22T01:01:56.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128036.log 2024-09-22T01:01:56.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91798.log 2024-09-22T01:01:56.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128036.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128036.log.gz 2024-09-22T01:01:56.389 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34240.log 2024-09-22T01:01:56.389 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91798.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141570.log 2024-09-22T01:01:56.390 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78213.log.gz 2024-09-22T01:01:56.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34240.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108769.log.gz 2024-09-22T01:01:56.390 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42010.log 2024-09-22T01:01:56.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.141570.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80735.log 2024-09-22T01:01:56.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42010.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36428.log 2024-09-22T01:01:56.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80735.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105478.log 2024-09-22T01:01:56.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36428.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108381.log 2024-09-22T01:01:56.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105478.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102366.log 2024-09-22T01:01:56.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.108381.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94546.log 2024-09-22T01:01:56.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58555.log 2024-09-22T01:01:56.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94546.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53216.log 2024-09-22T01:01:56.394 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68346.log.gz 2024-09-22T01:01:56.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58555.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37730.log 2024-09-22T01:01:56.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.53216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135379.log 2024-09-22T01:01:56.395 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37730.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68096.log 2024-09-22T01:01:56.395 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141570.log.gz 2024-09-22T01:01:56.395 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127015.log 2024-09-22T01:01:56.396 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125021.log 2024-09-22T01:01:56.396 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127015.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74692.log 2024-09-22T01:01:56.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42496.log 2024-09-22T01:01:56.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.74692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91307.log 2024-09-22T01:01:56.398 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42496.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122306.log 2024-09-22T01:01:56.398 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76353.log 2024-09-22T01:01:56.399 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117220.log 2024-09-22T01:01:56.399 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91000.log 2024-09-22T01:01:56.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.117220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58327.log 2024-09-22T01:01:56.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91000.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63096.log 2024-09-22T01:01:56.401 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58327.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85610.log 2024-09-22T01:01:56.401 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.63096.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63096.log.gz 2024-09-22T01:01:56.401 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139143.log 2024-09-22T01:01:56.402 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.85812.log.gz 2024-09-22T01:01:56.402 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.85610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93420.log 2024-09-22T01:01:56.402 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139143.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.139143.log.gz 2024-09-22T01:01:56.403 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63882.log 2024-09-22T01:01:56.403 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93420.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101107.log.gz 2024-09-22T01:01:56.404 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51371.log 2024-09-22T01:01:56.404 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111076.log 2024-09-22T01:01:56.404 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73565.log 2024-09-22T01:01:56.404 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101170.log.gz 2024-09-22T01:01:56.405 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111076.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104006.log 2024-09-22T01:01:56.405 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73565.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96041.log 2024-09-22T01:01:56.405 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104006.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75470.log 2024-09-22T01:01:56.405 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96041.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36007.log 2024-09-22T01:01:56.406 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75470.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124496.log 2024-09-22T01:01:56.406 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36007.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78170.log 2024-09-22T01:01:56.407 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124496.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62245.log 2024-09-22T01:01:56.407 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78170.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43538.log 2024-09-22T01:01:56.407 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112141.log.gz 2024-09-22T01:01:56.407 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62245.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65665.log 2024-09-22T01:01:56.408 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43538.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132599.log 2024-09-22T01:01:56.408 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65665.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44258.log 2024-09-22T01:01:56.409 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132599.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58808.log 2024-09-22T01:01:56.409 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105850.log 2024-09-22T01:01:56.410 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58808.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131285.log 2024-09-22T01:01:56.410 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80357.log 2024-09-22T01:01:56.410 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.131285.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120734.log 2024-09-22T01:01:56.410 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80357.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114972.log 2024-09-22T01:01:56.411 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120734.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88331.log 2024-09-22T01:01:56.411 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89993.log 2024-09-22T01:01:56.412 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88331.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63613.log 2024-09-22T01:01:56.412 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111929.log 2024-09-22T01:01:56.413 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.63613.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49748.log 2024-09-22T01:01:56.413 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111929.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70093.log 2024-09-22T01:01:56.413 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.49748.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.49748.log.gz 2024-09-22T01:01:56.414 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70093.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.33400.log.gz 2024-09-22T01:01:56.414 INFO:teuthology.orchestra.run.smithi060.stderr:gzip 90.1% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.33610.log.gz --verbose 2024-09-22T01:01:56.414 INFO:teuthology.orchestra.run.smithi060.stderr: -- /var/log/ceph/cluster1-client.mirror.45728.log 2024-09-22T01:01:56.415 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.33529.log.gz 2024-09-22T01:01:56.415 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.33528.log.gz 2024-09-22T01:01:56.415 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.33569.log.gz 2024-09-22T01:01:56.415 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129113.log 2024-09-22T01:01:56.416 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45728.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109636.log 2024-09-22T01:01:56.416 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129113.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108904.log 2024-09-22T01:01:56.417 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91084.log 2024-09-22T01:01:56.417 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.108904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70152.log 2024-09-22T01:01:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137312.log 2024-09-22T01:01:56.418 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126322.log.gz 2024-09-22T01:01:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86599.log 2024-09-22T01:01:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32228.log 2024-09-22T01:01:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.86599.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.29989.log.gz 2024-09-22T01:01:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88852.log 2024-09-22T01:01:56.419 INFO:teuthology.orchestra.run.smithi060.stderr: 58.7% -- replaced with /var/log/ceph/cluster1-client.mirror.30023.log.gz 2024-09-22T01:01:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32228.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.129258.log.gz 2024-09-22T01:01:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121010.log 90.1% 2024-09-22T01:01:56.420 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.127585.log.gz 2024-09-22T01:01:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88852.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73677.log 2024-09-22T01:01:56.422 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138130.log.gz 2024-09-22T01:01:56.422 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121010.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93143.log 2024-09-22T01:01:56.422 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73677.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133047.log 2024-09-22T01:01:56.422 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48673.log 2024-09-22T01:01:56.422 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133047.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112561.log 2024-09-22T01:01:56.425 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48673.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110394.log 2024-09-22T01:01:56.426 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112561.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.95201.log.gz 2024-09-22T01:01:56.426 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129115.log 2024-09-22T01:01:56.426 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110394.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104847.log 2024-09-22T01:01:56.426 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129115.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62050.log 2024-09-22T01:01:56.426 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104847.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48127.log 2024-09-22T01:01:56.426 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62050.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90689.log 2024-09-22T01:01:56.426 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48127.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54146.log 2024-09-22T01:01:56.426 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90689.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111017.log 2024-09-22T01:01:56.429 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54146.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61204.log 2024-09-22T01:01:56.429 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111017.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64880.log.gz 2024-09-22T01:01:56.429 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132030.log 2024-09-22T01:01:56.430 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61204.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109989.log 2024-09-22T01:01:56.430 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.61204.log.gz 2024-09-22T01:01:56.430 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72103.log 2024-09-22T01:01:56.430 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61566.log 2024-09-22T01:01:56.430 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72103.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.72103.log.gz 2024-09-22T01:01:56.430 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44508.log 2024-09-22T01:01:56.430 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61566.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116297.log 2024-09-22T01:01:56.430 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44508.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121535.log 2024-09-22T01:01:56.433 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116297.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53243.log 2024-09-22T01:01:56.433 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121535.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121115.log 2024-09-22T01:01:56.433 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.53243.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91299.log 2024-09-22T01:01:56.433 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121115.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138532.log.gz 2024-09-22T01:01:56.433 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138574.log.gz 2024-09-22T01:01:56.433 INFO:teuthology.orchestra.run.smithi060.stderr: /var/log/ceph/cluster1-client.mirror.110611.log 2024-09-22T01:01:56.434 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115559.log 2024-09-22T01:01:56.434 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86268.log 2024-09-22T01:01:56.434 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51963.log 2024-09-22T01:01:56.434 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.86268.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101570.log 2024-09-22T01:01:56.436 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139937.log 2024-09-22T01:01:56.436 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101570.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105741.log 2024-09-22T01:01:56.436 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139937.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105652.log 2024-09-22T01:01:56.436 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137020.log 2024-09-22T01:01:56.436 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.105652.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105652.log.gz 2024-09-22T01:01:56.436 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32372.log 2024-09-22T01:01:56.436 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45810.log.gz 2024-09-22T01:01:56.438 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137020.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38246.log 2024-09-22T01:01:56.438 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32372.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118386.log 2024-09-22T01:01:56.438 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38246.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48966.log 2024-09-22T01:01:56.438 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38246.log.gz 2024-09-22T01:01:56.438 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118386.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97350.log 2024-09-22T01:01:56.438 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48966.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60194.log 2024-09-22T01:01:56.440 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.97350.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86542.log 2024-09-22T01:01:56.440 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42563.log 2024-09-22T01:01:56.441 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.86542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114887.log 2024-09-22T01:01:56.441 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.42563.log: 68.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63882.log.gz 2024-09-22T01:01:56.441 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74578.log 2024-09-22T01:01:56.441 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114887.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113401.log.gz 2024-09-22T01:01:56.441 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57674.log 2024-09-22T01:01:56.443 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.74578.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114241.log 2024-09-22T01:01:56.443 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139365.log 2024-09-22T01:01:56.443 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114241.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90579.log 2024-09-22T01:01:56.443 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.139365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133991.log 2024-09-22T01:01:56.443 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90579.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63134.log 2024-09-22T01:01:56.443 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133991.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50330.log 2024-09-22T01:01:56.444 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.63134.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63134.log.gz 2024-09-22T01:01:56.444 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124139.log 2024-09-22T01:01:56.444 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50330.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108313.log 2024-09-22T01:01:56.444 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124139.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138367.log.gz 2024-09-22T01:01:56.444 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112831.log 2024-09-22T01:01:56.445 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.108313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50443.log 2024-09-22T01:01:56.445 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134371.log 2024-09-22T01:01:56.446 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61485.log 2024-09-22T01:01:56.446 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134371.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133532.log.gz 2024-09-22T01:01:56.446 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106263.log 2024-09-22T01:01:56.447 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61485.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66918.log 2024-09-22T01:01:56.447 INFO:teuthology.orchestra.run.smithi060.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.53243.log.gz 2024-09-22T01:01:56.447 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106263.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132449.log 2024-09-22T01:01:56.447 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80483.log 2024-09-22T01:01:56.448 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132449.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133012.log 2024-09-22T01:01:56.449 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83510.log 2024-09-22T01:01:56.449 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110432.log 2024-09-22T01:01:56.450 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65603.log 2024-09-22T01:01:56.450 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110432.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98640.log 2024-09-22T01:01:56.450 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86382.log 2024-09-22T01:01:56.451 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98640.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97832.log 2024-09-22T01:01:56.451 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.86382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128708.log 2024-09-22T01:01:56.451 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.97832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60445.log 2024-09-22T01:01:56.452 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128708.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136472.log 2024-09-22T01:01:56.452 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60445.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64992.log 2024-09-22T01:01:56.452 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113609.log.gz 2024-09-22T01:01:56.453 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136472.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70868.log 2024-09-22T01:01:56.454 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65471.log 2024-09-22T01:01:56.454 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111989.log 2024-09-22T01:01:56.455 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101442.log 2024-09-22T01:01:56.455 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103418.log 2024-09-22T01:01:56.455 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91948.log 2024-09-22T01:01:56.456 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.33568.log.gz 2024-09-22T01:01:56.456 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103418.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140358.log 2024-09-22T01:01:56.456 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91948.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44340.log 2024-09-22T01:01:56.457 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140358.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68976.log 2024-09-22T01:01:56.457 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70933.log 2024-09-22T01:01:56.457 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68976.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79136.log 2024-09-22T01:01:56.458 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70933.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68305.log 2024-09-22T01:01:56.458 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125022.log 2024-09-22T01:01:56.459 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68305.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46783.log 2024-09-22T01:01:56.459 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125022.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70363.log 2024-09-22T01:01:56.460 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46783.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141933.log 2024-09-22T01:01:56.460 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70363.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67047.log 2024-09-22T01:01:56.461 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.141933.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35982.log.gz 2024-09-22T01:01:56.461 INFO:teuthology.orchestra.run.smithi060.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.141933.log.gz 2024-09-22T01:01:56.461 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29055.log 2024-09-22T01:01:56.461 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36007.log.gz 2024-09-22T01:01:56.462 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67047.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78128.log 2024-09-22T01:01:56.462 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29055.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29055.log.gz 2024-09-22T01:01:56.462 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30864.log 2024-09-22T01:01:56.463 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110142.log 2024-09-22T01:01:56.463 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30864.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114661.log 2024-09-22T01:01:56.464 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110142.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92892.log 2024-09-22T01:01:56.464 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114661.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.40361.log.gz 2024-09-22T01:01:56.464 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.40366.log.gz 2024-09-22T01:01:56.464 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78127.log 2024-09-22T01:01:56.465 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92892.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.50330.log.gz 2024-09-22T01:01:56.465 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93208.log 2024-09-22T01:01:56.465 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78127.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44046.log 2024-09-22T01:01:56.465 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124183.log 2024-09-22T01:01:56.466 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44046.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66582.log 2024-09-22T01:01:56.466 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97759.log 2024-09-22T01:01:56.467 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66582.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140917.log 2024-09-22T01:01:56.467 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.97759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113399.log 2024-09-22T01:01:56.468 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140917.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64089.log 2024-09-22T01:01:56.468 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49836.log 2024-09-22T01:01:56.469 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115308.log 2024-09-22T01:01:56.469 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.49836.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.49836.log.gz 2024-09-22T01:01:56.469 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142047.log 2024-09-22T01:01:56.470 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115308.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44803.log.gz 2024-09-22T01:01:56.470 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114174.log 2024-09-22T01:01:56.470 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.142047.log: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.142047.log.gz 2024-09-22T01:01:56.471 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126073.log 2024-09-22T01:01:56.471 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83081.log 2024-09-22T01:01:56.472 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127331.log 2024-09-22T01:01:56.472 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83081.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.83081.log.gz 2024-09-22T01:01:56.472 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84612.log 2024-09-22T01:01:56.472 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116792.log 2024-09-22T01:01:56.473 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84612.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33319.log 2024-09-22T01:01:56.473 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32707.log.gz 2024-09-22T01:01:56.474 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.116792.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.67463.log.gz 2024-09-22T01:01:56.474 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.116792.log.gz 2024-09-22T01:01:56.474 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49287.log 2024-09-22T01:01:56.475 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33319.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67716.log 2024-09-22T01:01:56.475 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49287.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83793.log 2024-09-22T01:01:56.475 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69732.log 2024-09-22T01:01:56.476 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93210.log 2024-09-22T01:01:56.476 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69732.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125693.log 2024-09-22T01:01:56.476 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54146.log.gz 2024-09-22T01:01:56.476 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138868.log 2024-09-22T01:01:56.477 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125693.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138576.log 2024-09-22T01:01:56.477 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39407.log 2024-09-22T01:01:56.478 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75631.log 2024-09-22T01:01:56.478 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39407.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118936.log 2024-09-22T01:01:56.478 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75631.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65799.log.gz 2024-09-22T01:01:56.479 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139772.log 2024-09-22T01:01:56.479 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.118936.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60700.log 2024-09-22T01:01:56.479 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139772.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108749.log 2024-09-22T01:01:56.480 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90184.log 2024-09-22T01:01:56.480 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.108749.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78486.log 2024-09-22T01:01:56.481 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90184.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58683.log 2024-09-22T01:01:56.481 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78486.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96831.log 2024-09-22T01:01:56.481 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58683.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90413.log 2024-09-22T01:01:56.482 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96831.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96831.log.gz 2024-09-22T01:01:56.482 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38855.log 2024-09-22T01:01:56.483 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83206.log 2024-09-22T01:01:56.483 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.45961.log.gz 2024-09-22T01:01:56.483 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38855.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53082.log 2024-09-22T01:01:56.484 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.83206.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128997.log 2024-09-22T01:01:56.484 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53082.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35584.log 2024-09-22T01:01:56.485 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128997.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50441.log 2024-09-22T01:01:56.485 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35584.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68202.log 2024-09-22T01:01:56.486 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50441.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53329.log 2024-09-22T01:01:56.486 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68202.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45959.log.gz 2024-09-22T01:01:56.486 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100730.log 2024-09-22T01:01:56.486 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53329.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70193.log 2024-09-22T01:01:56.487 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100730.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89193.log 2024-09-22T01:01:56.487 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97095.log 2024-09-22T01:01:56.488 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128600.log 2024-09-22T01:01:56.488 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70093.log.gz 2024-09-22T01:01:56.488 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.97095.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44720.log 2024-09-22T01:01:56.489 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128600.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138826.log 2024-09-22T01:01:56.489 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45139.log 2024-09-22T01:01:56.490 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138826.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78379.log 2024-09-22T01:01:56.490 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45139.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118016.log 2024-09-22T01:01:56.491 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70934.log 2024-09-22T01:01:56.491 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79641.log.gz 2024-09-22T01:01:56.491 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118016.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70908.log 2024-09-22T01:01:56.492 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70934.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50820.log 2024-09-22T01:01:56.492 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57506.log 2024-09-22T01:01:56.493 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.50820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61467.log 2024-09-22T01:01:56.493 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57506.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53061.log 2024-09-22T01:01:56.494 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61467.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142322.log 2024-09-22T01:01:56.494 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.53061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37269.log 2024-09-22T01:01:56.495 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.142322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88792.log 2024-09-22T01:01:56.495 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37269.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.admin.142322.log.gz 2024-09-22T01:01:56.495 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56962.log 2024-09-22T01:01:56.495 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.82497.log.gz 2024-09-22T01:01:56.496 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102433.log 2024-09-22T01:01:56.496 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56962.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112140.log 2024-09-22T01:01:56.497 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102433.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43392.log 2024-09-22T01:01:56.497 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112140.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140359.log 2024-09-22T01:01:56.498 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43392.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62752.log 2024-09-22T01:01:56.498 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140359.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46908.log 2024-09-22T01:01:56.499 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58934.log 2024-09-22T01:01:56.499 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101907.log 2024-09-22T01:01:56.499 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58934.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135587.log 2024-09-22T01:01:56.500 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101907.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116274.log.gz 2024-09-22T01:01:56.500 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94175.log 2024-09-22T01:01:56.501 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135587.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49289.log 2024-09-22T01:01:56.501 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94175.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36299.log 2024-09-22T01:01:56.501 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113334.log 2024-09-22T01:01:56.502 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90268.log 2024-09-22T01:01:56.502 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113334.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118712.log 2024-09-22T01:01:56.503 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90268.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82161.log 2024-09-22T01:01:56.503 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.118712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52646.log 2024-09-22T01:01:56.504 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82161.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113502.log 2024-09-22T01:01:56.504 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80649.log.gz 2024-09-22T01:01:56.504 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.52646.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44509.log 2024-09-22T01:01:56.505 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113502.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46210.log 2024-09-22T01:01:56.505 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71478.log 2024-09-22T01:01:56.506 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84019.log 2024-09-22T01:01:56.506 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73196.log 2024-09-22T01:01:56.506 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.84019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36342.log 2024-09-22T01:01:56.507 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73196.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63170.log 2024-09-22T01:01:56.507 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36342.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46146.log 2024-09-22T01:01:56.508 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91000.log.gz 2024-09-22T01:01:56.508 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.63170.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63170.log.gz 2024-09-22T01:01:56.508 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93920.log 2024-09-22T01:01:56.508 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46146.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77011.log 2024-09-22T01:01:56.509 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113040.log 2024-09-22T01:01:56.510 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77011.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75034.log 2024-09-22T01:01:56.510 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38853.log 2024-09-22T01:01:56.510 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.75034.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53289.log 2024-09-22T01:01:56.511 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38853.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132920.log 2024-09-22T01:01:56.511 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.53289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100604.log 2024-09-22T01:01:56.512 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103793.log 2024-09-22T01:01:56.512 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87025.log 2024-09-22T01:01:56.513 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138869.log 2024-09-22T01:01:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.87025.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134958.log 2024-09-22T01:01:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138869.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110947.log 2024-09-22T01:01:56.515 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134958.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37478.log 2024-09-22T01:01:56.515 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110947.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52532.log 2024-09-22T01:01:56.515 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37478.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80945.log 2024-09-22T01:01:56.516 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.52532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32112.log 2024-09-22T01:01:56.516 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81366.log 2024-09-22T01:01:56.517 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112200.log 2024-09-22T01:01:56.517 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.90182.log.gz 2024-09-22T01:01:56.517 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68834.log 2024-09-22T01:01:56.518 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112200.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140283.log 2024-09-22T01:01:56.518 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52874.log 2024-09-22T01:01:56.518 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127373.log 2024-09-22T01:01:56.519 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.52874.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80461.log 2024-09-22T01:01:56.519 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127373.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35441.log 2024-09-22T01:01:56.520 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80461.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55742.log 2024-09-22T01:01:56.520 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35441.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99166.log 2024-09-22T01:01:56.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68724.log 2024-09-22T01:01:56.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99166.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116974.log 2024-09-22T01:01:56.521 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.126848.log.gz 2024-09-22T01:01:56.521 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37856.log 2024-09-22T01:01:56.522 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116974.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46784.log 2024-09-22T01:01:56.522 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37856.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137668.log 2024-09-22T01:01:56.523 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46784.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50951.log 2024-09-22T01:01:56.523 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137668.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91798.log.gz 2024-09-22T01:01:56.523 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55912.log 2024-09-22T01:01:56.524 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.50951.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106527.log 2024-09-22T01:01:56.525 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57926.log 2024-09-22T01:01:56.525 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106527.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117695.log 2024-09-22T01:01:56.525 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57926.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89531.log 2024-09-22T01:01:56.526 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.117695.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109377.log 2024-09-22T01:01:56.526 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89531.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127162.log 2024-09-22T01:01:56.527 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109377.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78485.log 2024-09-22T01:01:56.527 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127162.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87467.log 2024-09-22T01:01:56.527 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78485.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55867.log 2024-09-22T01:01:56.528 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.87467.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69673.log 2024-09-22T01:01:56.528 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55867.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102366.log.gz 2024-09-22T01:01:56.528 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137399.log 2024-09-22T01:01:56.529 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69673.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110461.log 2024-09-22T01:01:56.529 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102850.log 2024-09-22T01:01:56.530 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110461.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.88852.log.gz 2024-09-22T01:01:56.530 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66006.log 2024-09-22T01:01:56.530 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138742.log 2024-09-22T01:01:56.531 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66006.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59944.log 2024-09-22T01:01:56.532 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66306.log 2024-09-22T01:01:56.532 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.88942.log.gz 2024-09-22T01:01:56.532 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88766.log 2024-09-22T01:01:56.533 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111357.log 2024-09-22T01:01:56.533 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121637.log 2024-09-22T01:01:56.533 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111357.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40821.log 2024-09-22T01:01:56.534 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128032.log 2024-09-22T01:01:56.534 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40821.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40821.log.gz 2024-09-22T01:01:56.534 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75335.log 2024-09-22T01:01:56.535 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128032.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128032.log.gz 2024-09-22T01:01:56.535 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.140111.log 2024-09-22T01:01:56.536 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.75335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77101.log 2024-09-22T01:01:56.536 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.140111.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95544.log 2024-09-22T01:01:56.536 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49371.log.gz 2024-09-22T01:01:56.537 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115289.log 2024-09-22T01:01:56.537 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.95544.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98190.log 2024-09-22T01:01:56.537 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56792.log 2024-09-22T01:01:56.538 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.98190.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69044.log 2024-09-22T01:01:56.538 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104950.log 2024-09-22T01:01:56.539 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134957.log 2024-09-22T01:01:56.539 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.132276.log.gz 2024-09-22T01:01:56.539 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104950.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55402.log 2024-09-22T01:01:56.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134957.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123866.log 2024-09-22T01:01:56.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55402.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61988.log 2024-09-22T01:01:56.540 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123866.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126952.log 2024-09-22T01:01:56.541 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.61988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35629.log 2024-09-22T01:01:56.541 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126952.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133309.log 2024-09-22T01:01:56.542 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35629.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74423.log 2024-09-22T01:01:56.542 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133309.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141371.log 2024-09-22T01:01:56.543 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.74423.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138448.log 2024-09-22T01:01:56.543 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47915.log.gz 2024-09-22T01:01:56.543 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.141371.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141371.log.gz 2024-09-22T01:01:56.543 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89304.log 2024-09-22T01:01:56.544 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138448.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106369.log 2024-09-22T01:01:56.544 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89304.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48924.log 2024-09-22T01:01:56.545 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104007.log 2024-09-22T01:01:56.545 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48924.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78297.log 2024-09-22T01:01:56.546 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104007.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136807.log 2024-09-22T01:01:56.546 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78297.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54470.log 2024-09-22T01:01:56.547 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136807.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123321.log 2024-09-22T01:01:56.547 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54470.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105308.log 2024-09-22T01:01:56.548 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72503.log 2024-09-22T01:01:56.548 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105308.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90539.log 2024-09-22T01:01:56.549 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35627.log 2024-09-22T01:01:56.549 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90539.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63654.log 2024-09-22T01:01:56.550 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35627.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104111.log 2024-09-22T01:01:56.550 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63654.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30143.log 2024-09-22T01:01:56.550 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104111.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48841.log 2024-09-22T01:01:56.551 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.30143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98449.log 2024-09-22T01:01:56.551 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48841.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36889.log 2024-09-22T01:01:56.552 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.98449.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117940.log 2024-09-22T01:01:56.553 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.97759.log.gz 2024-09-22T01:01:56.553 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36889.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140501.log 2024-09-22T01:01:56.553 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.117940.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.111989.log.gz 2024-09-22T01:01:56.553 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56289.log 2024-09-22T01:01:56.554 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.140501.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120772.log 2024-09-22T01:01:56.554 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104951.log 2024-09-22T01:01:56.555 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39213.log 2024-09-22T01:01:56.555 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115308.log.gz 2024-09-22T01:01:56.555 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104951.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107940.log 2024-09-22T01:01:56.555 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130943.log 2024-09-22T01:01:56.556 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107940.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.107940.log.gz 2024-09-22T01:01:56.556 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99887.log 2024-09-22T01:01:56.557 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.130943.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138576.log.gz 2024-09-22T01:01:56.557 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65768.log 2024-09-22T01:01:56.558 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106687.log 2024-09-22T01:01:56.558 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65768.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32271.log 2024-09-22T01:01:56.559 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70427.log 2024-09-22T01:01:56.559 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132684.log 2024-09-22T01:01:56.559 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70427.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124601.log 2024-09-22T01:01:56.560 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132684.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70404.log 2024-09-22T01:01:56.560 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50373.log 2024-09-22T01:01:56.560 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70404.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56121.log 2024-09-22T01:01:56.562 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50373.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53155.log 2024-09-22T01:01:56.562 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136763.log 2024-09-22T01:01:56.562 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.53155.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70194.log 2024-09-22T01:01:56.562 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109636.log.gz 2024-09-22T01:01:56.565 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136763.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72894.log 2024-09-22T01:01:56.565 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123720.log 2024-09-22T01:01:56.565 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65434.log 2024-09-22T01:01:56.565 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115246.log 2024-09-22T01:01:56.565 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65434.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127331.log.gz 2024-09-22T01:01:56.565 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120798.log 2024-09-22T01:01:56.565 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115246.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115709.log 2024-09-22T01:01:56.567 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120798.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81867.log 2024-09-22T01:01:56.567 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115709.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43560.log 2024-09-22T01:01:56.567 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81867.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114661.log.gz 2024-09-22T01:01:56.567 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65305.log 2024-09-22T01:01:56.568 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43560.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140906.log 2024-09-22T01:01:56.568 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65305.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59376.log 2024-09-22T01:01:56.568 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102324.log 2024-09-22T01:01:56.569 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77962.log 2024-09-22T01:01:56.569 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102324.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39961.log 2024-09-22T01:01:56.569 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77962.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39616.log 2024-09-22T01:01:56.570 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112831.log.gz 2024-09-22T01:01:56.570 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39961.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113082.log 2024-09-22T01:01:56.570 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39616.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40107.log 2024-09-22T01:01:56.571 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113082.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45643.log 2024-09-22T01:01:56.571 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40107.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82877.log 2024-09-22T01:01:56.572 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140173.log 2024-09-22T01:01:56.572 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112727.log 2024-09-22T01:01:56.572 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140173.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91625.log 2024-09-22T01:01:56.573 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112727.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107988.log 2024-09-22T01:01:56.574 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91625.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97692.log 2024-09-22T01:01:56.574 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107988.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.107988.log.gz 2024-09-22T01:01:56.574 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58976.log 2024-09-22T01:01:56.575 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.97692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124979.log 2024-09-22T01:01:56.575 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58976.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82246.log 2024-09-22T01:01:56.576 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124979.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.129114.log.gz 2024-09-22T01:01:56.576 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131191.log 2024-09-22T01:01:56.576 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82246.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64130.log 2024-09-22T01:01:56.576 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.131191.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119632.log 2024-09-22T01:01:56.577 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.64130.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95772.log 2024-09-22T01:01:56.577 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.119632.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33169.log 2024-09-22T01:01:56.578 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.95772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138658.log 2024-09-22T01:01:56.578 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33169.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71748.log 2024-09-22T01:01:56.579 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138658.log: 87.5%gzip -- replaced with /var/log/ceph/cluster1-client.admin.61485.log.gz -5 2024-09-22T01:01:56.579 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.100602.log 2024-09-22T01:01:56.579 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71748.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46188.log 2024-09-22T01:01:56.580 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75758.log 2024-09-22T01:01:56.580 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65540.log 2024-09-22T01:01:56.581 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75758.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45266.log 2024-09-22T01:01:56.581 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.65540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112621.log 2024-09-22T01:01:56.582 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45266.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78843.log 2024-09-22T01:01:56.582 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112621.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30949.log 2024-09-22T01:01:56.582 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78843.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68413.log 2024-09-22T01:01:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.30949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80607.log 2024-09-22T01:01:56.583 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.98110.log.gz 2024-09-22T01:01:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42817.log 2024-09-22T01:01:56.584 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80607.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63587.log 2024-09-22T01:01:56.584 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42817.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100017.log 2024-09-22T01:01:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.63587.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31537.log 2024-09-22T01:01:56.585 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.107098.log.gz 2024-09-22T01:01:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100017.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37984.log 2024-09-22T01:01:56.585 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.130943.log.gz 2024-09-22T01:01:56.586 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.31537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125652.log 2024-09-22T01:01:56.586 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.37984.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55657.log 2024-09-22T01:01:56.587 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125652.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50818.log 2024-09-22T01:01:56.587 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55657.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77716.log 2024-09-22T01:01:56.588 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.50818.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101487.log 2024-09-22T01:01:56.588 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92303.log 2024-09-22T01:01:56.589 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91298.log 2024-09-22T01:01:56.589 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105741.log.gz 2024-09-22T01:01:56.589 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45687.log 2024-09-22T01:01:56.590 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91298.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105850.log.gz 2024-09-22T01:01:56.590 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123592.log 2024-09-22T01:01:56.590 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45687.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141456.log 2024-09-22T01:01:56.591 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37792.log 2024-09-22T01:01:56.591 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.141456.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48377.log 2024-09-22T01:01:56.591 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54617.log 2024-09-22T01:01:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48377.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.30335.log 2024-09-22T01:01:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54617.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141550.log 2024-09-22T01:01:56.593 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.0.30335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34870.log 2024-09-22T01:01:56.593 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.141550.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98556.log 2024-09-22T01:01:56.594 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34870.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141550.log.gz 2024-09-22T01:01:56.594 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68018.log 2024-09-22T01:01:56.596 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98556.log: 87.4% 87.6%gzip -- replaced with /var/log/ceph/cluster2-client.admin.140359.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.140358.log.gz 2024-09-22T01:01:56.596 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:01:56.596 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93729.log 2024-09-22T01:01:56.596 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49643.log 2024-09-22T01:01:56.596 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93729.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141456.log.gz 2024-09-22T01:01:56.597 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93629.log 2024-09-22T01:01:56.597 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.49643.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.49643.log.gz 2024-09-22T01:01:56.597 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114132.log 2024-09-22T01:01:56.598 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93629.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49008.log 2024-09-22T01:01:56.598 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77338.log 2024-09-22T01:01:56.599 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49008.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138827.log 2024-09-22T01:01:56.599 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77338.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66792.log 2024-09-22T01:01:56.599 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138827.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33907.log 2024-09-22T01:01:56.600 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109209.log 2024-09-22T01:01:56.600 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33907.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59711.log.gz 2024-09-22T01:01:56.600 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142546.log 2024-09-22T01:01:56.601 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.109209.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33697.log 2024-09-22T01:01:56.602 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.142546.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118018.log 2024-09-22T01:01:56.602 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142546.log.gz 2024-09-22T01:01:56.602 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33697.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117696.log 2024-09-22T01:01:56.602 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94458.log 2024-09-22T01:01:56.603 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.117696.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96385.log 2024-09-22T01:01:56.603 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.94458.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113292.log 2024-09-22T01:01:56.604 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94458.log.gz 2024-09-22T01:01:56.604 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96385.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136033.log 2024-09-22T01:01:56.604 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113292.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79053.log 2024-09-22T01:01:56.604 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64520.log 2024-09-22T01:01:56.605 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79053.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68513.log 2024-09-22T01:01:56.606 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64520.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113081.log 2024-09-22T01:01:56.606 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68513.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131098.log 2024-09-22T01:01:56.607 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122501.log 2024-09-22T01:01:56.607 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131098.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31176.log 2024-09-22T01:01:56.607 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122501.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99433.log 2024-09-22T01:01:56.608 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31176.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87130.log 2024-09-22T01:01:56.608 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99433.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35442.log 2024-09-22T01:01:56.609 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87130.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115224.log 2024-09-22T01:01:56.609 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103333.log 2024-09-22T01:01:56.610 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115224.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105098.log 2024-09-22T01:01:56.610 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81995.log 2024-09-22T01:01:56.611 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105098.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42689.log 2024-09-22T01:01:56.611 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81995.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.30297.log 2024-09-22T01:01:56.611 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42689.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74806.log 2024-09-22T01:01:56.612 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.0.30297.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49894.log 2024-09-22T01:01:56.612 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.74806.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44003.log 2024-09-22T01:01:56.613 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.49894.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.49894.log.gz 2024-09-22T01:01:56.613 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105619.log 2024-09-22T01:01:56.613 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112979.log 2024-09-22T01:01:56.614 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.105619.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105619.log.gz 2024-09-22T01:01:56.614 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69608.log 2024-09-22T01:01:56.614 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112979.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90622.log 2024-09-22T01:01:56.615 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126177.log 2024-09-22T01:01:56.615 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90622.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29591.log 2024-09-22T01:01:56.616 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103629.log 2024-09-22T01:01:56.616 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.29591.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29591.log.gz 2024-09-22T01:01:56.617 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131564.log 2024-09-22T01:01:56.617 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103629.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106777.log 2024-09-22T01:01:56.617 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36929.log 2024-09-22T01:01:56.617 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.106777.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55576.log.gz 2024-09-22T01:01:56.618 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85651.log 2024-09-22T01:01:56.618 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36929.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122614.log 2024-09-22T01:01:56.619 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.85651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33462.log 2024-09-22T01:01:56.619 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122614.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37033.log 2024-09-22T01:01:56.620 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33462.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111721.log 2024-09-22T01:01:56.620 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124644.log 2024-09-22T01:01:56.620 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111721.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50819.log 2024-09-22T01:01:56.621 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111130.log 2024-09-22T01:01:56.621 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.110016.log.gz 2024-09-22T01:01:56.622 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.50819.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97624.log 2024-09-22T01:01:56.622 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111130.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30640.log 2024-09-22T01:01:56.623 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.97624.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131326.log 2024-09-22T01:01:56.623 INFO:teuthology.orchestra.run.smithi060.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.admin.72894.log.gz 2024-09-22T01:01:56.623 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30640.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.73196.log.gz 2024-09-22T01:01:56.623 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85041.log 2024-09-22T01:01:56.623 INFO:teuthology.orchestra.run.smithi060.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63654.log.gz 2024-09-22T01:01:56.624 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131326.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60277.log 2024-09-22T01:01:56.624 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.85041.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124074.log 2024-09-22T01:01:56.625 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60277.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41799.log 2024-09-22T01:01:56.625 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124074.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111612.log 2024-09-22T01:01:56.626 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41799.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142254.log 2024-09-22T01:01:56.626 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111612.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43259.log 2024-09-22T01:01:56.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.142254.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119538.log 2024-09-22T01:01:56.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86724.log 2024-09-22T01:01:56.627 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.119538.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108407.log 2024-09-22T01:01:56.628 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.86724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95610.log 2024-09-22T01:01:56.628 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.108407.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101148.log 2024-09-22T01:01:56.629 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.95610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76416.log 2024-09-22T01:01:56.629 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101148.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91612.log 2024-09-22T01:01:56.630 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118466.log 2024-09-22T01:01:56.631 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91612.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59943.log 2024-09-22T01:01:56.631 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.51751.log.gz 2024-09-22T01:01:56.631 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.118466.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67338.log 2024-09-22T01:01:56.632 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59943.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37993.log 2024-09-22T01:01:56.632 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67338.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134453.log 2024-09-22T01:01:56.633 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.37993.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30211.log 2024-09-22T01:01:56.633 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134453.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.142254.log.gz 2024-09-22T01:01:56.633 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139202.log 2024-09-22T01:01:56.633 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.30211.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71563.log 2024-09-22T01:01:56.634 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139202.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.139202.log.gz 2024-09-22T01:01:56.634 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125711.log 2024-09-22T01:01:56.634 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71563.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137061.log 2024-09-22T01:01:56.635 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125711.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101736.log 2024-09-22T01:01:56.635 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101653.log 2024-09-22T01:01:56.636 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101736.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100309.log 2024-09-22T01:01:56.636 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116676.log 2024-09-22T01:01:56.637 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44021.log 2024-09-22T01:01:56.637 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42817.log.gz 2024-09-22T01:01:56.638 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126032.log 2024-09-22T01:01:56.638 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44021.log: /var/log/ceph/cluster1-client.mirror.116676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38091.log 2024-09-22T01:01:56.638 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126032.log: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.47749.log.gz 2024-09-22T01:01:56.638 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88984.log 2024-09-22T01:01:56.639 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.38091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88891.log 2024-09-22T01:01:56.639 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88984.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89909.log 2024-09-22T01:01:56.640 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88891.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41746.log 2024-09-22T01:01:56.640 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89909.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83339.log 2024-09-22T01:01:56.641 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.41746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135520.log 2024-09-22T01:01:56.641 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83339.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.83339.log.gz 2024-09-22T01:01:56.641 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89363.log 2024-09-22T01:01:56.642 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135520.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111016.log 2024-09-22T01:01:56.642 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89363.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108883.log 2024-09-22T01:01:56.643 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111016.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96758.log 2024-09-22T01:01:56.643 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.108883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80315.log 2024-09-22T01:01:56.643 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96758.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96758.log.gz 2024-09-22T01:01:56.644 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58851.log 2024-09-22T01:01:56.644 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80315.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61468.log 2024-09-22T01:01:56.645 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58851.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28985.log 2024-09-22T01:01:56.645 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61468.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35081.log 2024-09-22T01:01:56.645 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.28985.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28985.log.gz 2024-09-22T01:01:56.646 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115878.log 2024-09-22T01:01:56.646 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128895.log 2024-09-22T01:01:56.646 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115878.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132534.log 2024-09-22T01:01:56.647 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55276.log 2024-09-22T01:01:56.648 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132534.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104174.log 2024-09-22T01:01:56.648 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55276.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62751.log 2024-09-22T01:01:56.649 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105980.log 2024-09-22T01:01:56.649 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112349.log 2024-09-22T01:01:56.650 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.105980.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55868.log 2024-09-22T01:01:56.650 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112349.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50709.log 2024-09-22T01:01:56.650 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55868.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130963.log 2024-09-22T01:01:56.651 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50709.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122543.log 2024-09-22T01:01:56.651 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.130963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.140064.log 2024-09-22T01:01:56.652 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122543.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116484.log 2024-09-22T01:01:56.653 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.140064.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134557.log 2024-09-22T01:01:56.653 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116484.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71353.log 2024-09-22T01:01:56.654 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134557.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95991.log 2024-09-22T01:01:56.654 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82667.log 2024-09-22T01:01:56.654 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.95991.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36931.log 2024-09-22T01:01:56.655 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134328.log 2024-09-22T01:01:56.655 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61069.log 2024-09-22T01:01:56.655 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53555.log.gz 2024-09-22T01:01:56.656 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134328.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85952.log 2024-09-22T01:01:56.656 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61069.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122669.log 2024-09-22T01:01:56.657 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.85952.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65906.log 2024-09-22T01:01:56.657 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122669.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42267.log 2024-09-22T01:01:56.657 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72071.log 2024-09-22T01:01:56.658 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42267.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71773.log 2024-09-22T01:01:56.659 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72071.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.72071.log.gz 2024-09-22T01:01:56.659 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76479.log 2024-09-22T01:01:56.659 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71773.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116929.log 2024-09-22T01:01:56.660 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76479.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63841.log 2024-09-22T01:01:56.660 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.116929.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.116929.log.gz 2024-09-22T01:01:56.660 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38918.log 2024-09-22T01:01:56.661 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.63841.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109293.log 2024-09-22T01:01:56.662 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85671.log 2024-09-22T01:01:56.662 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109293.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94469.log 2024-09-22T01:01:56.662 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.85671.log: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.52787.log.gz 2024-09-22T01:01:56.663 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100436.log 2024-09-22T01:01:56.663 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94469.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132093.log 2024-09-22T01:01:56.663 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100436.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34031.log 2024-09-22T01:01:56.664 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63359.log 2024-09-22T01:01:56.664 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96961.log 2024-09-22T01:01:56.665 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.63359.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121046.log 2024-09-22T01:01:56.665 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.96961.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31496.log 2024-09-22T01:01:56.666 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121046.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78883.log 2024-09-22T01:01:56.666 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31496.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102996.log 2024-09-22T01:01:56.667 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70866.log 2024-09-22T01:01:56.667 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102996.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48335.log 2024-09-22T01:01:56.667 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70866.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58221.log 2024-09-22T01:01:56.668 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130461.log 2024-09-22T01:01:56.668 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32457.log 2024-09-22T01:01:56.669 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.130461.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125442.log 2024-09-22T01:01:56.669 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32457.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137905.log 2024-09-22T01:01:56.670 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123236.log 2024-09-22T01:01:56.671 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42237.log 2024-09-22T01:01:56.671 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123236.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56121.log.gz 2024-09-22T01:01:56.671 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72082.log 2024-09-22T01:01:56.671 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42237.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69648.log 2024-09-22T01:01:56.672 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.72082.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31140.log 2024-09-22T01:01:56.672 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69648.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47161.log 2024-09-22T01:01:56.673 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110117.log 2024-09-22T01:01:56.673 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47161.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58387.log 2024-09-22T01:01:56.674 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110117.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140803.log 2024-09-22T01:01:56.675 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58387.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94938.log 2024-09-22T01:01:56.675 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140803.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129683.log 2024-09-22T01:01:56.675 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.94938.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140273.log 2024-09-22T01:01:56.675 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129683.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31433.log 2024-09-22T01:01:56.676 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135419.log 2024-09-22T01:01:56.676 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31433.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44214.log 2024-09-22T01:01:56.677 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135419.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100813.log 2024-09-22T01:01:56.678 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44214.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101234.log 2024-09-22T01:01:56.678 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100813.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123213.log 2024-09-22T01:01:56.679 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101234.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62053.log 2024-09-22T01:01:56.679 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123213.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141591.log 2024-09-22T01:01:56.679 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58683.log.gz 2024-09-22T01:01:56.680 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62053.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82897.log 2024-09-22T01:01:56.680 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.141591.log: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.141591.log.gz 2024-09-22T01:01:56.680 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87870.log 2024-09-22T01:01:56.681 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82897.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89303.log 2024-09-22T01:01:56.681 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87870.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86688.log 2024-09-22T01:01:56.682 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46908.log.gz 2024-09-22T01:01:56.682 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123594.log 2024-09-22T01:01:56.682 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.86688.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34512.log 2024-09-22T01:01:56.683 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123594.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102851.log 2024-09-22T01:01:56.683 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64520.log.gz 2024-09-22T01:01:56.683 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34512.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96843.log 2024-09-22T01:01:56.684 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41659.log 2024-09-22T01:01:56.684 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96843.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96843.log.gz 2024-09-22T01:01:56.684 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76859.log 2024-09-22T01:01:56.685 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.41659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69356.log 2024-09-22T01:01:56.686 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76859.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136175.log 2024-09-22T01:01:56.686 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46784.log.gz 2024-09-22T01:01:56.686 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69356.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70092.log 2024-09-22T01:01:56.687 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136175.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50764.log 2024-09-22T01:01:56.687 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70092.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65433.log 2024-09-22T01:01:56.688 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50764.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109776.log 2024-09-22T01:01:56.689 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65433.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89010.log 2024-09-22T01:01:56.689 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109776.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59440.log 2024-09-22T01:01:56.689 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89010.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35711.log 2024-09-22T01:01:56.690 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59440.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98428.log 2024-09-22T01:01:56.691 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35711.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70193.log.gz 2024-09-22T01:01:56.691 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92010.log 2024-09-22T01:01:56.691 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70194.log.gz 2024-09-22T01:01:56.691 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98428.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61251.log 2024-09-22T01:01:56.692 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92010.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41632.log 2024-09-22T01:01:56.692 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137231.log 2024-09-22T01:01:56.693 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.41632.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86766.log 2024-09-22T01:01:56.693 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137231.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96387.log 2024-09-22T01:01:56.694 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.86766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111509.log 2024-09-22T01:01:56.694 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96387.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81238.log 2024-09-22T01:01:56.695 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41431.log 2024-09-22T01:01:56.696 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81238.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69673.log.gz 2024-09-22T01:01:56.696 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81111.log 2024-09-22T01:01:56.696 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.41431.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69648.log.gz 2024-09-22T01:01:56.696 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111930.log 2024-09-22T01:01:56.697 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50948.log 2024-09-22T01:01:56.698 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111930.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126281.log 2024-09-22T01:01:56.698 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.50948.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126113.log 2024-09-22T01:01:56.699 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126281.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114299.log 2024-09-22T01:01:56.699 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126113.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74879.log 2024-09-22T01:01:56.700 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92912.log 2024-09-22T01:01:56.700 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.74879.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67548.log 2024-09-22T01:01:56.700 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92912.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62052.log 2024-09-22T01:01:56.701 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113567.log 2024-09-22T01:01:56.701 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62052.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45139.log.gz 2024-09-22T01:01:56.702 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30545.log 2024-09-22T01:01:56.702 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113567.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42458.log 2024-09-22T01:01:56.702 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.30545.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96644.log 2024-09-22T01:01:56.704 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42458.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121155.log 2024-09-22T01:01:56.704 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79810.log 2024-09-22T01:01:56.705 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121155.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87721.log 2024-09-22T01:01:56.705 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79810.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92302.log 2024-09-22T01:01:56.706 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87721.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37602.log 2024-09-22T01:01:56.706 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92302.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100962.log 2024-09-22T01:01:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37602.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84226.log 2024-09-22T01:01:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100962.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93099.log 2024-09-22T01:01:56.708 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58514.log 2024-09-22T01:01:56.708 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93099.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120376.log 2024-09-22T01:01:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55401.log 2024-09-22T01:01:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128229.log 2024-09-22T01:01:56.710 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55401.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87615.log 2024-09-22T01:01:56.710 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.128229.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127793.log 2024-09-22T01:01:56.711 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.87615.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109038.log 2024-09-22T01:01:56.711 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127793.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100813.log.gz 2024-09-22T01:01:56.712 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78004.log 2024-09-22T01:01:56.712 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.109038.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125126.log 2024-09-22T01:01:56.712 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91569.log 2024-09-22T01:01:56.713 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125126.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68018.log.gz 2024-09-22T01:01:56.713 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120015.log 2024-09-22T01:01:56.714 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68096.log.gz 2024-09-22T01:01:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91569.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68054.log.gz 2024-09-22T01:01:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74290.log 2024-09-22T01:01:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120015.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102324.log.gz 2024-09-22T01:01:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133178.log 2024-09-22T01:01:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.74290.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74290.log.gz 2024-09-22T01:01:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65880.log 2024-09-22T01:01:56.716 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133178.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135212.log 2024-09-22T01:01:56.717 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74920.log 2024-09-22T01:01:56.718 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135212.log: /var/log/ceph/cluster1-client.admin.74920.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116676.log.gz 2024-09-22T01:01:56.718 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140275.log 2024-09-22T01:01:56.719 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61068.log 2024-09-22T01:01:56.720 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134832.log 2024-09-22T01:01:56.720 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61068.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.61068.log.gz 2024-09-22T01:01:56.720 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129457.log 2024-09-22T01:01:56.721 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134832.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97054.log 2024-09-22T01:01:56.721 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.129457.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34198.log 2024-09-22T01:01:56.722 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.97054.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71706.log 2024-09-22T01:01:56.722 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34198.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54789.log 2024-09-22T01:01:56.723 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71706.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90898.log.gz 2024-09-22T01:01:56.723 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68623.log 2024-09-22T01:01:56.724 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54789.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66368.log 2024-09-22T01:01:56.724 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68623.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59228.log 2024-09-22T01:01:56.725 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66368.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67573.log 2024-09-22T01:01:56.725 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59228.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125126.log.gz 2024-09-22T01:01:56.725 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107759.log 2024-09-22T01:01:56.725 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67573.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87659.log 2024-09-22T01:01:56.726 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103903.log 2024-09-22T01:01:56.727 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41545.log 2024-09-22T01:01:56.727 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126323.log.gz 2024-09-22T01:01:56.728 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103903.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66666.log 2024-09-22T01:01:56.728 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.41545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111720.log 2024-09-22T01:01:56.729 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66666.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37350.log 2024-09-22T01:01:56.729 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111720.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.30640.log.gz 2024-09-22T01:01:56.729 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101023.log 2024-09-22T01:01:56.729 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.104007.log.gz 2024-09-22T01:01:56.730 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112074.log 2024-09-22T01:01:56.730 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101023.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.105308.log.gz 2024-09-22T01:01:56.731 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116235.log 2024-09-22T01:01:56.731 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112074.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83318.log 2024-09-22T01:01:56.732 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116235.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135420.log 2024-09-22T01:01:56.732 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.83318.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82667.log.gz 2024-09-22T01:01:56.732 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86180.log 2024-09-22T01:01:56.733 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135420.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136850.log 2024-09-22T01:01:56.733 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.86180.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116130.log 2024-09-22T01:01:56.734 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115769.log 2024-09-22T01:01:56.734 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116130.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133092.log 2024-09-22T01:01:56.735 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57591.log 2024-09-22T01:01:56.735 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111509.log.gz 2024-09-22T01:01:56.735 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133092.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107750.log 2024-09-22T01:01:56.736 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57591.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50570.log 2024-09-22T01:01:56.737 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107750.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76602.log 2024-09-22T01:01:56.737 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.82897.log.gz 2024-09-22T01:01:56.737 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50570.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126804.log 2024-09-22T01:01:56.738 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126324.log 2024-09-22T01:01:56.738 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126804.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120609.log 2024-09-22T01:01:56.739 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126324.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52854.log 2024-09-22T01:01:56.739 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120609.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43840.log 2024-09-22T01:01:56.740 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52854.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50821.log 2024-09-22T01:01:56.740 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126952.log.gz 2024-09-22T01:01:56.741 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43840.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98998.log 2024-09-22T01:01:56.741 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.50821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120692.log 2024-09-22T01:01:56.741 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98998.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127686.log 2024-09-22T01:01:56.742 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.120692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98237.log 2024-09-22T01:01:56.742 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.102220.log.gz 2024-09-22T01:01:56.743 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127686.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29289.log 2024-09-22T01:01:56.743 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98237.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85879.log 2024-09-22T01:01:56.744 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29289.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29289.log.gz 2024-09-22T01:01:56.744 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103209.log 2024-09-22T01:01:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.85879.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119492.log 2024-09-22T01:01:56.745 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124496.log.gz 2024-09-22T01:01:56.745 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103209.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51739.log 2024-09-22T01:01:56.746 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.119492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80210.log 2024-09-22T01:01:56.746 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126491.log 2024-09-22T01:01:56.747 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92260.log 2024-09-22T01:01:56.747 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63110.log 2024-09-22T01:01:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44721.log 2024-09-22T01:01:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.63110.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63110.log.gz 2024-09-22T01:01:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62505.log 2024-09-22T01:01:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44721.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121685.log 2024-09-22T01:01:56.750 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.62505.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128702.log 2024-09-22T01:01:56.750 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121685.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93858.log 2024-09-22T01:01:56.750 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128702.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92303.log.gz 2024-09-22T01:01:56.751 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137987.log 2024-09-22T01:01:56.751 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93858.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48128.log 2024-09-22T01:01:56.752 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137987.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108153.log 2024-09-22T01:01:56.752 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113400.log 2024-09-22T01:01:56.753 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.108153.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87361.log 2024-09-22T01:01:56.753 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.48673.log.gz 2024-09-22T01:01:56.754 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108448.log 2024-09-22T01:01:56.754 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87361.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49242.log 2024-09-22T01:01:56.755 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.108448.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101276.log 2024-09-22T01:01:56.755 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49242.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112662.log 2024-09-22T01:01:56.756 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101276.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138996.log 2024-09-22T01:01:56.756 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112662.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133367.log 2024-09-22T01:01:56.757 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138996.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101171.log 2024-09-22T01:01:56.757 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124644.log.gz 2024-09-22T01:01:56.758 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133367.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127122.log 2024-09-22T01:01:56.758 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101171.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103542.log 2024-09-22T01:01:56.759 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109786.log 2024-09-22T01:01:56.759 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142161.log 2024-09-22T01:01:56.760 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109786.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92369.log 2024-09-22T01:01:56.760 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136175.log.gz 2024-09-22T01:01:56.761 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.142161.log: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.142161.log.gz 2024-09-22T01:01:56.761 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71477.log 2024-09-22T01:01:56.761 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123719.log 2024-09-22T01:01:56.762 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78925.log 2024-09-22T01:01:56.762 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123719.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104803.log 2024-09-22T01:01:56.763 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78925.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102221.log 2024-09-22T01:01:56.763 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.92892.log.gz 2024-09-22T01:01:56.764 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104803.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97209.log 2024-09-22T01:01:56.764 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80860.log 2024-09-22T01:01:56.765 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.97209.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.140065.log.gz 2024-09-22T01:01:56.765 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124225.log 2024-09-22T01:01:56.765 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135212.log.gz 2024-09-22T01:01:56.766 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80860.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61497.log 2024-09-22T01:01:56.766 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124225.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84419.log 2024-09-22T01:01:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61497.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44088.log 2024-09-22T01:01:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84419.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73258.log 2024-09-22T01:01:56.768 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45349.log 2024-09-22T01:01:56.768 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132684.log.gz 2024-09-22T01:01:56.769 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.73258.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128229.log.gz 2024-09-22T01:01:56.769 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39075.log 2024-09-22T01:01:56.769 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45349.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108065.log 2024-09-22T01:01:56.770 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39075.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31471.log 2024-09-22T01:01:56.771 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.108065.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119199.log 2024-09-22T01:01:56.771 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31471.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119652.log 2024-09-22T01:01:56.772 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.119199.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119199.log.gz 2024-09-22T01:01:56.772 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104636.log 2024-09-22T01:01:56.772 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.119652.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.137103.log.gz 2024-09-22T01:01:56.772 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33359.log 2024-09-22T01:01:56.773 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104636.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84225.log 2024-09-22T01:01:56.773 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33359.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117588.log 2024-09-22T01:01:56.774 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.107759.log.gz 2024-09-22T01:01:56.774 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114407.log 2024-09-22T01:01:56.774 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.117588.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.61323.log.gz 2024-09-22T01:01:56.774 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.137231.log.gz 2024-09-22T01:01:56.775 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119163.log 2024-09-22T01:01:56.775 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61251.log.gz 2024-09-22T01:01:56.775 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114407.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125861.log 2024-09-22T01:01:56.776 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.119163.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119163.log.gz 2024-09-22T01:01:56.776 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134078.log 2024-09-22T01:01:56.777 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125861.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44003.log.gz 2024-09-22T01:01:56.777 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63446.log 2024-09-22T01:01:56.778 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134078.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52626.log 2024-09-22T01:01:56.778 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.63446.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105350.log 2024-09-22T01:01:56.778 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122752.log 2024-09-22T01:01:56.779 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131753.log 2024-09-22T01:01:56.780 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122752.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73655.log 2024-09-22T01:01:56.780 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55868.log.gz 2024-09-22T01:01:56.781 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131753.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96567.log 2024-09-22T01:01:56.781 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43516.log 2024-09-22T01:01:56.781 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.96567.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39173.log 2024-09-22T01:01:56.782 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43516.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85858.log 2024-09-22T01:01:56.783 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39173.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47285.log 2024-09-22T01:01:56.783 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.85858.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137905.log.gz 2024-09-22T01:01:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84329.log 2024-09-22T01:01:56.784 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47285.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120518.log 2024-09-22T01:01:56.785 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.84329.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89698.log 2024-09-22T01:01:56.785 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120518.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119424.log 2024-09-22T01:01:56.786 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70109.log 2024-09-22T01:01:56.786 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.119424.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85331.log 2024-09-22T01:01:56.787 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70109.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81910.log 2024-09-22T01:01:56.787 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.85331.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85331.log.gz 2024-09-22T01:01:56.788 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125764.log 2024-09-22T01:01:56.788 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81910.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98043.log 2024-09-22T01:01:56.789 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125764.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56876.log 2024-09-22T01:01:56.789 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90900.log 2024-09-22T01:01:56.790 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56876.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67783.log 2024-09-22T01:01:56.790 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90900.log: gzip 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134832.log.gz -5 2024-09-22T01:01:56.790 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.34660.log 2024-09-22T01:01:56.791 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67783.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43908.log 2024-09-22T01:01:56.791 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34660.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133564.log 2024-09-22T01:01:56.792 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43908.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28829.log 2024-09-22T01:01:56.792 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133564.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119187.log 2024-09-22T01:01:56.793 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.28829.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28829.log.gz 2024-09-22T01:01:56.793 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135757.log 2024-09-22T01:01:56.793 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.119187.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119187.log.gz 2024-09-22T01:01:56.794 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78757.log 2024-09-22T01:01:56.795 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47748.log 2024-09-22T01:01:56.795 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36553.log 2024-09-22T01:01:56.796 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47748.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41115.log 2024-09-22T01:01:56.796 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36553.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42369.log 2024-09-22T01:01:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41115.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44173.log 2024-09-22T01:01:56.797 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62505.log.gz 2024-09-22T01:01:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113987.log 2024-09-22T01:01:56.798 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44173.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130248.log 2024-09-22T01:01:56.799 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113987.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73205.log 2024-09-22T01:01:56.799 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.130248.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130248.log.gz 2024-09-22T01:01:56.799 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64926.log 2024-09-22T01:01:56.800 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73205.log: 90.0%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.133178.log.gz -5 2024-09-22T01:01:56.800 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.75201.log 2024-09-22T01:01:56.800 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64926.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129902.log 2024-09-22T01:01:56.801 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.75201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34722.log 2024-09-22T01:01:56.802 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.129902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47327.log 2024-09-22T01:01:56.802 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.83511.log.gz 2024-09-22T01:01:56.802 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37985.log 2024-09-22T01:01:56.803 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47327.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62268.log 2024-09-22T01:01:56.803 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.37985.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51396.log 2024-09-22T01:01:56.804 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62268.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45140.log 2024-09-22T01:01:56.804 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119606.log 2024-09-22T01:01:56.805 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45140.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141892.log 2024-09-22T01:01:56.805 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.119606.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58303.log 2024-09-22T01:01:56.806 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.141892.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.141892.log.gz 2024-09-22T01:01:56.806 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96855.log 2024-09-22T01:01:56.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70237.log 2024-09-22T01:01:56.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96855.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96855.log.gz 2024-09-22T01:01:56.807 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102010.log 2024-09-22T01:01:56.808 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70237.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85493.log 2024-09-22T01:01:56.808 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102010.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133950.log 2024-09-22T01:01:56.809 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.85493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66454.log 2024-09-22T01:01:56.809 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66666.log.gz 2024-09-22T01:01:56.810 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133950.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89678.log 2024-09-22T01:01:56.810 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131902.log 2024-09-22T01:01:56.811 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89678.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108977.log 2024-09-22T01:01:56.811 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.131902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113252.log 2024-09-22T01:01:56.811 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67548.log.gz 2024-09-22T01:01:56.812 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.108977.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80691.log.gz 2024-09-22T01:01:56.812 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142791.log 2024-09-22T01:01:56.813 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113252.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51398.log 2024-09-22T01:01:56.813 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.142791.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142791.log.gz 2024-09-22T01:01:56.813 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127372.log 2024-09-22T01:01:56.814 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114240.log 2024-09-22T01:01:56.815 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127372.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68894.log 2024-09-22T01:01:56.815 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114240.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64110.log 2024-09-22T01:01:56.815 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87659.log.gz 2024-09-22T01:01:56.816 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50578.log 2024-09-22T01:01:56.816 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105796.log 2024-09-22T01:01:56.817 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50578.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92370.log 2024-09-22T01:01:56.817 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105796.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53999.log 2024-09-22T01:01:56.817 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.89429.log.gz 2024-09-22T01:01:56.818 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92370.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102010.log.gz 2024-09-22T01:01:56.818 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127836.log 2024-09-22T01:01:56.819 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53999.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47202.log 2024-09-22T01:01:56.819 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127836.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29616.log 2024-09-22T01:01:56.820 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47202.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86522.log 2024-09-22T01:01:56.820 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.29616.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29616.log.gz 2024-09-22T01:01:56.821 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131145.log 2024-09-22T01:01:56.821 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.86522.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66407.log 2024-09-22T01:01:56.822 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.131145.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90412.log 2024-09-22T01:01:56.822 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66407.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129573.log 2024-09-22T01:01:56.823 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90412.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47075.log 2024-09-22T01:01:56.823 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129573.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64800.log 2024-09-22T01:01:56.824 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47075.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104530.log 2024-09-22T01:01:56.824 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.64800.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68623.log.gz 2024-09-22T01:01:56.825 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129529.log 2024-09-22T01:01:56.825 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104530.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111676.log 2024-09-22T01:01:56.826 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.129529.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113190.log 2024-09-22T01:01:56.827 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57382.log 2024-09-22T01:01:56.827 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113190.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46825.log 2024-09-22T01:01:56.827 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136387.log 2024-09-22T01:01:56.828 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46825.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78673.log 2024-09-22T01:01:56.828 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136387.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117782.log 2024-09-22T01:01:56.829 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78673.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107511.log 2024-09-22T01:01:56.830 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.117782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60404.log 2024-09-22T01:01:56.830 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48798.log 2024-09-22T01:01:56.831 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60404.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63028.log 2024-09-22T01:01:56.831 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48798.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86469.log 2024-09-22T01:01:56.832 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.63028.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115876.log 2024-09-22T01:01:56.832 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.86469.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75315.log 2024-09-22T01:01:56.833 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115876.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132877.log 2024-09-22T01:01:56.833 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.75315.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104321.log 2024-09-22T01:01:56.834 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40947.log 2024-09-22T01:01:56.834 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122604.log 2024-09-22T01:01:56.835 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.40947.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63768.log 2024-09-22T01:01:56.835 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124707.log 2024-09-22T01:01:56.836 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.63768.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64394.log 2024-09-22T01:01:56.836 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124707.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44021.log.gz 2024-09-22T01:01:56.837 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52304.log 2024-09-22T01:01:56.837 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64394.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54362.log 2024-09-22T01:01:56.838 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.52304.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92578.log 2024-09-22T01:01:56.838 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44088.log.gz 2024-09-22T01:01:56.838 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.54362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78443.log 2024-09-22T01:01:56.839 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92578.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116235.log.gz 2024-09-22T01:01:56.839 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66453.log 2024-09-22T01:01:56.840 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31348.log 2024-09-22T01:01:56.840 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35167.log 2024-09-22T01:01:56.841 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96475.log 2024-09-22T01:01:56.842 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35167.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138658.log.gz 2024-09-22T01:01:56.842 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141912.log 2024-09-22T01:01:56.842 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96475.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137270.log 2024-09-22T01:01:56.843 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.141912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109931.log 2024-09-22T01:01:56.843 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104298.log 2024-09-22T01:01:56.844 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86640.log 2024-09-22T01:01:56.844 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104298.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107252.log 2024-09-22T01:01:56.845 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.86640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89638.log 2024-09-22T01:01:56.845 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107252.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139880.log 2024-09-22T01:01:56.845 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122604.log.gz 2024-09-22T01:01:56.846 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89638.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100182.log 2024-09-22T01:01:56.847 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76498.log 2024-09-22T01:01:56.848 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100182.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116130.log.gz 2024-09-22T01:01:56.848 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123131.log 2024-09-22T01:01:56.848 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80817.log 2024-09-22T01:01:56.848 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141912.log.gz 2024-09-22T01:01:56.849 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61018.log 2024-09-22T01:01:56.850 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80817.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92260.log.gz 2024-09-22T01:01:56.850 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76566.log 2024-09-22T01:01:56.850 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61018.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.61018.log.gz 2024-09-22T01:01:56.850 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44970.log 2024-09-22T01:01:56.851 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76566.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86107.log 2024-09-22T01:01:56.852 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44970.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128714.log 2024-09-22T01:01:56.852 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126324.log.gz 2024-09-22T01:01:56.852 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.86107.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52833.log 2024-09-22T01:01:56.853 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128714.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89740.log 2024-09-22T01:01:56.854 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64271.log 2024-09-22T01:01:56.854 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100017.log.gz 2024-09-22T01:01:56.854 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72329.log 2024-09-22T01:01:56.855 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.64271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81049.log 2024-09-22T01:01:56.855 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72329.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.72329.log.gz 2024-09-22T01:01:56.855 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131057.log 2024-09-22T01:01:56.856 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81049.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70616.log 2024-09-22T01:01:56.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.131057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123890.log 2024-09-22T01:01:56.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99886.log 2024-09-22T01:01:56.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123890.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44172.log 2024-09-22T01:01:56.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49454.log 2024-09-22T01:01:56.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44172.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102229.log 2024-09-22T01:01:56.859 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103542.log.gz 2024-09-22T01:01:56.860 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131622.log 2024-09-22T01:01:56.860 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102229.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45895.log 2024-09-22T01:01:56.861 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131622.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77628.log 2024-09-22T01:01:56.861 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45895.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51754.log 2024-09-22T01:01:56.862 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77628.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75107.log 2024-09-22T01:01:56.863 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51754.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107105.log 2024-09-22T01:01:56.863 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.75107.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142570.log 2024-09-22T01:01:56.863 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.107105.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80062.log 2024-09-22T01:01:56.864 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.142570.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.142570.log.gz 2024-09-22T01:01:56.864 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99119.log 2024-09-22T01:01:56.865 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127372.log.gz 2024-09-22T01:01:56.865 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88044.log 2024-09-22T01:01:56.865 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105796.log.gz 2024-09-22T01:01:56.866 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99119.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83794.log 2024-09-22T01:01:56.867 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120035.log 2024-09-22T01:01:56.867 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83794.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.129902.log.gz 2024-09-22T01:01:56.867 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56435.log 2024-09-22T01:01:56.868 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.120035.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117138.log 2024-09-22T01:01:56.868 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.112662.log.gz 2024-09-22T01:01:56.868 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56435.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119720.log 2024-09-22T01:01:56.869 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.117138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37995.log 2024-09-22T01:01:56.870 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.119720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52398.log 2024-09-22T01:01:56.871 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.37995.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78547.log 2024-09-22T01:01:56.871 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.84225.log.gz 2024-09-22T01:01:56.871 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124392.log 2024-09-22T01:01:56.871 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78547.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30126.log 2024-09-22T01:01:56.872 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124392.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122021.log 2024-09-22T01:01:56.872 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.30126.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29063.log 2024-09-22T01:01:56.873 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57129.log 2024-09-22T01:01:56.873 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29063.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29063.log.gz 2024-09-22T01:01:56.874 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78003.log 2024-09-22T01:01:56.874 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57129.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123696.log 2024-09-22T01:01:56.875 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33672.log 2024-09-22T01:01:56.875 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123696.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102159.log 2024-09-22T01:01:56.876 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33672.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111284.log 2024-09-22T01:01:56.876 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105162.log 2024-09-22T01:01:56.877 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111284.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73357.log 2024-09-22T01:01:56.877 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105162.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119860.log 2024-09-22T01:01:56.878 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.73357.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89950.log 2024-09-22T01:01:56.879 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.119860.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30177.log 2024-09-22T01:01:56.879 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89950.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135128.log 2024-09-22T01:01:56.880 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.30177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55319.log 2024-09-22T01:01:56.880 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72947.log 2024-09-22T01:01:56.881 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55319.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136895.log 2024-09-22T01:01:56.881 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72947.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80440.log 2024-09-22T01:01:56.882 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93838.log 2024-09-22T01:01:56.883 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80440.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41404.log 2024-09-22T01:01:56.883 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34241.log 2024-09-22T01:01:56.883 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.41404.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100541.log 2024-09-22T01:01:56.884 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34241.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56497.log 2024-09-22T01:01:56.884 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42534.log 2024-09-22T01:01:56.885 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56497.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100393.log 2024-09-22T01:01:56.885 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42534.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136512.log 2024-09-22T01:01:56.886 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100393.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126241.log 2024-09-22T01:01:56.887 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49815.log 2024-09-22T01:01:56.887 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126241.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89152.log 2024-09-22T01:01:56.887 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49815.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126596.log 2024-09-22T01:01:56.888 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37560.log 2024-09-22T01:01:56.889 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126596.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58851.log.gz 2024-09-22T01:01:56.889 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106385.log 2024-09-22T01:01:56.890 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37560.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107124.log 2024-09-22T01:01:56.890 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106385.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68472.log 2024-09-22T01:01:56.891 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.107124.log: 68.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52398.log.gz 2024-09-22T01:01:56.891 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117570.log 2024-09-22T01:01:56.891 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.52304.log.gz 2024-09-22T01:01:56.891 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68472.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37689.log 2024-09-22T01:01:56.892 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.117570.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96807.log 2024-09-22T01:01:56.893 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37689.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46020.log 2024-09-22T01:01:56.893 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96807.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96807.log.gz 2024-09-22T01:01:56.894 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131077.log 2024-09-22T01:01:56.894 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46020.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31618.log 2024-09-22T01:01:56.895 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.131077.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41860.log 2024-09-22T01:01:56.895 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31618.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124140.log 2024-09-22T01:01:56.896 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.41860.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78819.log 2024-09-22T01:01:56.897 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114510.log 2024-09-22T01:01:56.897 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.109038.log.gz 2024-09-22T01:01:56.897 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85584.log 2024-09-22T01:01:56.898 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131685.log 2024-09-22T01:01:56.898 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.85584.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137819.log 2024-09-22T01:01:56.899 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131685.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68977.log 2024-09-22T01:01:56.899 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120376.log.gz 2024-09-22T01:01:56.900 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98006.log 2024-09-22T01:01:56.900 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68977.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78694.log 2024-09-22T01:01:56.901 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98006.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.120518.log.gz 2024-09-22T01:01:56.901 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63674.log 2024-09-22T01:01:56.901 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78694.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.120798.log.gz 2024-09-22T01:01:56.902 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85392.log 2024-09-22T01:01:56.902 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.63674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116549.log 2024-09-22T01:01:56.903 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.85392.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85392.log.gz 2024-09-22T01:01:56.903 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34242.log 2024-09-22T01:01:56.904 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116549.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116633.log 2024-09-22T01:01:56.905 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34242.log: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.64110.log.gz 2024-09-22T01:01:56.905 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120645.log 2024-09-22T01:01:56.905 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116633.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130284.log 2024-09-22T01:01:56.906 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117876.log 2024-09-22T01:01:56.906 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.130284.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130284.log.gz 2024-09-22T01:01:56.906 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81113.log 2024-09-22T01:01:56.907 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.117876.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59605.log 2024-09-22T01:01:56.907 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81113.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132154.log 2024-09-22T01:01:56.908 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59605.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98869.log 2024-09-22T01:01:56.909 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132154.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108655.log 2024-09-22T01:01:56.909 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98869.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112769.log 2024-09-22T01:01:56.910 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.108655.log: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.131459.log.gz 2024-09-22T01:01:56.910 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76671.log 2024-09-22T01:01:56.910 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78214.log 2024-09-22T01:01:56.911 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.76671.log: 80.5% -- replaced with /var/log/ceph/cluster1-client.mirror.131564.log.gz 2024-09-22T01:01:56.911 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139723.log 2024-09-22T01:01:56.911 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78214.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95801.log 2024-09-22T01:01:56.912 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139723.log: gzip 87.8% -5 -- replaced with /var/log/ceph/cluster1-client.admin.131902.log.gz --verbose 2024-09-22T01:01:56.912 INFO:teuthology.orchestra.run.smithi060.stderr: -- /var/log/ceph/cluster1-client.mirror.70724.log 2024-09-22T01:01:56.913 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.95801.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29856.log 2024-09-22T01:01:56.913 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55442.log 2024-09-22T01:01:56.913 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.29856.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29856.log.gz 2024-09-22T01:01:56.914 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111075.log 2024-09-22T01:01:56.915 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96215.log 2024-09-22T01:01:56.915 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111075.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45390.log 2024-09-22T01:01:56.916 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.96215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81279.log 2024-09-22T01:01:56.916 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87511.log 2024-09-22T01:01:56.917 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81279.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48632.log 2024-09-22T01:01:56.917 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35587.log 2024-09-22T01:01:56.918 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48632.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81617.log 2024-09-22T01:01:56.918 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35587.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.33359.log.gz 2024-09-22T01:01:56.918 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92159.log 2024-09-22T01:01:56.919 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77492.log 2024-09-22T01:01:56.920 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74719.log 2024-09-22T01:01:56.920 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31348.log.gz 2024-09-22T01:01:56.920 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79513.log 2024-09-22T01:01:56.921 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.74719.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37099.log 2024-09-22T01:01:56.922 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79513.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137945.log 2024-09-22T01:01:56.922 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36553.log.gz 2024-09-22T01:01:56.922 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37099.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58177.log 2024-09-22T01:01:56.923 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135924.log 2024-09-22T01:01:56.924 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80566.log 2024-09-22T01:01:56.924 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135924.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47033.log 2024-09-22T01:01:56.925 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33672.log.gz 2024-09-22T01:01:56.925 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80566.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39415.log 2024-09-22T01:01:56.925 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88374.log 2024-09-22T01:01:56.926 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39415.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29137.log 2024-09-22T01:01:56.927 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68893.log 2024-09-22T01:01:56.927 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29137.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29137.log.gz 2024-09-22T01:01:56.927 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39405.log 2024-09-22T01:01:56.928 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68893.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97168.log 2024-09-22T01:01:56.928 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.86640.log.gz 2024-09-22T01:01:56.929 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39405.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116943.log 2024-09-22T01:01:56.929 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.38918.log.gz 2024-09-22T01:01:56.929 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.97168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41062.log 2024-09-22T01:01:56.930 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.116943.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.116943.log.gz 2024-09-22T01:01:56.930 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90329.log 2024-09-22T01:01:56.930 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.41062.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.31471.log.gz 2024-09-22T01:01:56.931 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123656.log 2024-09-22T01:01:56.932 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90329.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.50821.log.gz 2024-09-22T01:01:56.932 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126532.log 2024-09-22T01:01:56.932 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123656.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77627.log 2024-09-22T01:01:56.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117454.log 2024-09-22T01:01:56.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77627.log: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42458.log.gz 2024-09-22T01:01:56.934 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83561.log 2024-09-22T01:01:56.934 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.117454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133138.log 2024-09-22T01:01:56.935 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133789.log 2024-09-22T01:01:56.935 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114384.log 2024-09-22T01:01:56.936 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133789.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61104.log 2024-09-22T01:01:56.936 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72005.log 2024-09-22T01:01:56.937 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.61104.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130715.log 2024-09-22T01:01:56.937 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72005.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.72005.log.gz 2024-09-22T01:01:56.937 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105458.log 2024-09-22T01:01:56.938 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.130715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69942.log 2024-09-22T01:01:56.939 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105458.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74377.log 2024-09-22T01:01:56.939 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69942.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64715.log 2024-09-22T01:01:56.940 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.74377.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92469.log 2024-09-22T01:01:56.940 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50569.log 2024-09-22T01:01:56.941 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92469.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101024.log 2024-09-22T01:01:56.942 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50569.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134539.log 2024-09-22T01:01:56.942 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101024.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113148.log 2024-09-22T01:01:56.943 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134539.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128068.log 2024-09-22T01:01:56.943 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.54362.log.gz 2024-09-22T01:01:56.944 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113148.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73843.log 2024-09-22T01:01:56.944 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.128068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68263.log 2024-09-22T01:01:56.945 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73843.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42754.log 2024-09-22T01:01:56.945 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68263.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119579.log 2024-09-22T01:01:56.945 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0%/var/log/ceph/cluster1-client.mirror.42754.log: -- replaced with /var/log/ceph/cluster1-client.mirror.81049.log.gz 2024-09-22T01:01:56.946 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40797.log 2024-09-22T01:01:56.947 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.119579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68683.log 2024-09-22T01:01:56.947 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.41632.log.gz 2024-09-22T01:01:56.947 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40797.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40797.log.gz 2024-09-22T01:01:56.948 INFO:teuthology.orchestra.run.smithi060.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-client.admin.41659.log.gz 2024-09-22T01:01:56.948 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123066.log 2024-09-22T01:01:56.949 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68683.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49956.log 2024-09-22T01:01:56.949 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123066.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33738.log 2024-09-22T01:01:56.950 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.49956.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.49956.log.gz 2024-09-22T01:01:56.950 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96745.log 2024-09-22T01:01:56.950 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33738.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141778.log 2024-09-22T01:01:56.951 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96745.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96745.log.gz 2024-09-22T01:01:56.951 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43679.log 2024-09-22T01:01:56.952 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.141778.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141778.log.gz 2024-09-22T01:01:56.952 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68624.log 2024-09-22T01:01:56.953 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43679.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88001.log 2024-09-22T01:01:56.953 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68624.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40359.log 2024-09-22T01:01:56.954 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88001.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68977.log.gz 2024-09-22T01:01:56.954 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73642.log 2024-09-22T01:01:56.955 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.49815.log.gz 2024-09-22T01:01:56.955 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40359.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35398.log 2024-09-22T01:01:56.955 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121945.log 2024-09-22T01:01:56.956 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1%/var/log/ceph/cluster1-client.mirror.35398.log: -- replaced with /var/log/ceph/cluster1-client.mirror.70153.log.gz 2024-09-22T01:01:56.956 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60193.log 2024-09-22T01:01:56.956 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57718.log 2024-09-22T01:01:56.957 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116612.log 2024-09-22T01:01:56.958 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57718.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98596.log 2024-09-22T01:01:56.958 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116612.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64715.log.gz 2024-09-22T01:01:56.958 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136764.log 2024-09-22T01:01:56.959 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.98596.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102579.log 2024-09-22T01:01:56.959 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136764.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123931.log 2024-09-22T01:01:56.960 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53129.log 2024-09-22T01:01:56.960 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126985.log 2024-09-22T01:01:56.961 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.53129.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114317.log 2024-09-22T01:01:56.962 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115979.log 2024-09-22T01:01:56.962 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44804.log.gz 2024-09-22T01:01:56.962 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114317.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59901.log 2024-09-22T01:01:56.963 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115979.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101380.log 2024-09-22T01:01:56.964 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59901.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141953.log 2024-09-22T01:01:56.964 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.47202.log.gz 2024-09-22T01:01:56.964 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36112.log 2024-09-22T01:01:56.965 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.141953.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.141953.log.gz 2024-09-22T01:01:56.965 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80526.log 2024-09-22T01:01:56.966 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132196.log 2024-09-22T01:01:56.966 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80526.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.30306.log 2024-09-22T01:01:56.967 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.132196.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56626.log 2024-09-22T01:01:56.967 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68263.log.gz 2024-09-22T01:01:56.967 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.0.30306.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141299.log 2024-09-22T01:01:56.968 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36510.log 2024-09-22T01:01:56.969 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.141299.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.141299.log.gz 2024-09-22T01:01:56.969 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47685.log 2024-09-22T01:01:56.969 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77576.log 2024-09-22T01:01:56.970 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47685.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109699.log 2024-09-22T01:01:56.971 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67927.log 2024-09-22T01:01:56.971 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109699.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51065.log 2024-09-22T01:01:56.971 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68472.log.gz 2024-09-22T01:01:56.971 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67927.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37058.log 2024-09-22T01:01:56.972 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58326.log 2024-09-22T01:01:56.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37058.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115854.log 2024-09-22T01:01:56.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58326.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38092.log 2024-09-22T01:01:56.974 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115854.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69563.log 2024-09-22T01:01:56.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.38092.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62741.log 2024-09-22T01:01:56.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69563.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142026.log 2024-09-22T01:01:56.976 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105795.log 2024-09-22T01:01:56.976 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.142026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37392.log 2024-09-22T01:01:56.976 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.105795.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105795.log.gz 2024-09-22T01:01:56.977 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41176.log 2024-09-22T01:01:56.977 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37392.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134160.log 2024-09-22T01:01:56.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.41176.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53269.log 2024-09-22T01:01:56.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134160.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48631.log 2024-09-22T01:01:56.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.53269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71624.log 2024-09-22T01:01:56.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48631.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137776.log 2024-09-22T01:01:56.980 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71624.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90580.log 2024-09-22T01:01:56.980 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137776.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142482.log 2024-09-22T01:01:56.981 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.142026.log.gz 2024-09-22T01:01:56.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90580.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130890.log 2024-09-22T01:01:56.982 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.142482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70572.log 2024-09-22T01:01:56.982 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.130890.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107964.log 2024-09-22T01:01:56.983 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70572.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48086.log 2024-09-22T01:01:56.983 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107964.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.107964.log.gz 2024-09-22T01:01:56.983 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127541.log 2024-09-22T01:01:56.984 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48086.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44466.log 2024-09-22T01:01:56.985 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127541.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41229.log 2024-09-22T01:01:56.985 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44466.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36761.log 2024-09-22T01:01:56.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41229.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74266.log 2024-09-22T01:01:56.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36761.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.142482.log.gz 2024-09-22T01:01:56.986 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99083.log 2024-09-22T01:01:56.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.74266.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74266.log.gz 2024-09-22T01:01:56.987 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32791.log 2024-09-22T01:01:56.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99083.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124932.log 2024-09-22T01:01:56.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32791.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53689.log.gz 2024-09-22T01:01:56.988 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59712.log 2024-09-22T01:01:56.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124932.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75449.log 2024-09-22T01:01:56.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138406.log 2024-09-22T01:01:56.990 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.75449.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96043.log 2024-09-22T01:01:56.990 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138406.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63747.log 2024-09-22T01:01:56.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96043.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63975.log 2024-09-22T01:01:56.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.63747.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47162.log 2024-09-22T01:01:56.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.63975.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31435.log 2024-09-22T01:01:56.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47162.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140856.log 2024-09-22T01:01:56.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31435.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71202.log 2024-09-22T01:01:56.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.140856.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51785.log 2024-09-22T01:01:56.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71202.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123760.log 2024-09-22T01:01:56.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61020.log 2024-09-22T01:01:56.995 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123760.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126386.log 2024-09-22T01:01:56.995 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61020.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.61020.log.gz 2024-09-22T01:01:56.995 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39136.log 2024-09-22T01:01:56.996 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126386.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128703.log 2024-09-22T01:01:56.996 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55615.log 2024-09-22T01:01:56.997 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137566.log 2024-09-22T01:01:56.998 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55615.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53435.log 2024-09-22T01:01:56.998 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137566.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37603.log 2024-09-22T01:01:56.999 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53435.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137652.log 2024-09-22T01:01:56.999 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57800.log.gz 2024-09-22T01:01:56.999 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37603.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112769.log.gz 2024-09-22T01:01:56.999 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98743.log 2024-09-22T01:01:57.000 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59857.log 2024-09-22T01:01:57.001 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.98743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91949.log 2024-09-22T01:01:57.001 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59857.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33611.log 2024-09-22T01:01:57.002 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103166.log 2024-09-22T01:01:57.002 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116654.log 2024-09-22T01:01:57.003 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103166.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118263.log 2024-09-22T01:01:57.003 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116654.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91210.log 2024-09-22T01:01:57.004 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.118263.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116064.log 2024-09-22T01:01:57.005 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110081.log 2024-09-22T01:01:57.005 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116064.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57211.log.gz 2024-09-22T01:01:57.005 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88895.log 2024-09-22T01:01:57.005 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31202.log 2024-09-22T01:01:57.006 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93630.log 2024-09-22T01:01:57.007 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31202.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106585.log 2024-09-22T01:01:57.007 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93630.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73861.log 2024-09-22T01:01:57.007 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106585.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.49519.log.gz 2024-09-22T01:01:57.008 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32050.log 2024-09-22T01:01:57.008 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73861.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89428.log 2024-09-22T01:01:57.009 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32050.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123386.log 2024-09-22T01:01:57.009 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89428.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111464.log 2024-09-22T01:01:57.009 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123386.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79980.log 2024-09-22T01:01:57.010 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111464.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40090.log 2024-09-22T01:01:57.010 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79980.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36721.log 2024-09-22T01:01:57.011 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.40090.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31580.log 2024-09-22T01:01:57.012 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36721.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74066.log 2024-09-22T01:01:57.012 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31580.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64415.log 2024-09-22T01:01:57.012 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.74066.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140174.log 2024-09-22T01:01:57.013 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.64415.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120560.log 2024-09-22T01:01:57.014 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30487.log 2024-09-22T01:01:57.014 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.120560.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56289.log.gz 2024-09-22T01:01:57.014 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85044.log 2024-09-22T01:01:57.015 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137818.log 2024-09-22T01:01:57.015 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.85044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104508.log 2024-09-22T01:01:57.016 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137818.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125066.log 2024-09-22T01:01:57.016 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104508.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84831.log 2024-09-22T01:01:57.017 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125066.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76123.log 2024-09-22T01:01:57.017 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.84831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102493.log 2024-09-22T01:01:57.018 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76123.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60656.log 2024-09-22T01:01:57.018 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94407.log 2024-09-22T01:01:57.019 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60656.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119673.log 2024-09-22T01:01:57.019 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94407.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100773.log 2024-09-22T01:01:57.020 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.119673.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98168.log 2024-09-22T01:01:57.020 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100773.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136259.log 2024-09-22T01:01:57.021 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31897.log 2024-09-22T01:01:57.021 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105685.log 2024-09-22T01:01:57.022 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31897.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120222.log 2024-09-22T01:01:57.022 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.105685.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105685.log.gz 2024-09-22T01:01:57.023 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79768.log 2024-09-22T01:01:57.023 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.120222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81321.log 2024-09-22T01:01:57.024 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79768.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57591.log.gz 2024-09-22T01:01:57.024 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47328.log 2024-09-22T01:01:57.024 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121218.log 2024-09-22T01:01:57.025 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57925.log 2024-09-22T01:01:57.025 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121218.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140651.log 2024-09-22T01:01:57.026 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57925.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96150.log 2024-09-22T01:01:57.026 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.140651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105742.log 2024-09-22T01:01:57.027 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.96150.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75974.log 2024-09-22T01:01:57.027 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81660.log 2024-09-22T01:01:57.028 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75974.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103963.log 2024-09-22T01:01:57.028 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81660.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82415.log 2024-09-22T01:01:57.029 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113989.log 2024-09-22T01:01:57.029 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82415.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39373.log 2024-09-22T01:01:57.030 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84033.log 2024-09-22T01:01:57.030 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39373.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128011.log 2024-09-22T01:01:57.031 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.84033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79348.log 2024-09-22T01:01:57.031 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.128011.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91337.log 2024-09-22T01:01:57.031 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.48926.log.gz 2024-09-22T01:01:57.032 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79348.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122711.log 2024-09-22T01:01:57.032 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91337.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51401.log 2024-09-22T01:01:57.033 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122711.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54025.log 2024-09-22T01:01:57.033 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51401.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81575.log 2024-09-22T01:01:57.034 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54025.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116966.log 2024-09-22T01:01:57.034 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81575.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128300.log 2024-09-22T01:01:57.035 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116966.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39140.log 2024-09-22T01:01:57.035 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.128300.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142713.log 2024-09-22T01:01:57.036 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134411.log 2024-09-22T01:01:57.036 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.142713.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142713.log.gz 2024-09-22T01:01:57.036 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102535.log 2024-09-22T01:01:57.037 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134411.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76270.log 2024-09-22T01:01:57.038 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102535.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40711.log 2024-09-22T01:01:57.038 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138028.log 2024-09-22T01:01:57.038 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40711.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40711.log.gz 2024-09-22T01:01:57.039 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81195.log 2024-09-22T01:01:57.039 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138028.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51651.log 2024-09-22T01:01:57.040 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58222.log.gz 2024-09-22T01:01:57.040 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81195.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95346.log 2024-09-22T01:01:57.040 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78758.log 2024-09-22T01:01:57.041 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.95346.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59986.log 2024-09-22T01:01:57.041 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78758.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66624.log 2024-09-22T01:01:57.042 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59986.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112031.log 2024-09-22T01:01:57.042 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66624.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69564.log 2024-09-22T01:01:57.043 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65073.log 2024-09-22T01:01:57.043 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125612.log 2024-09-22T01:01:57.044 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65073.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72830.log 2024-09-22T01:01:57.044 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125612.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120393.log 2024-09-22T01:01:57.045 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.72830.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31836.log 2024-09-22T01:01:57.045 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120393.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58642.log 2024-09-22T01:01:57.046 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.31836.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78086.log 2024-09-22T01:01:57.046 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51183.log 2024-09-22T01:01:57.047 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78086.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.103691.log.gz 2024-09-22T01:01:57.047 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117127.log 2024-09-22T01:01:57.047 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.84831.log.gz 2024-09-22T01:01:57.048 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.51183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67003.log 2024-09-22T01:01:57.048 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.117127.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.117127.log.gz 2024-09-22T01:01:57.048 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56434.log 2024-09-22T01:01:57.049 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125231.log 2024-09-22T01:01:57.049 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.88374.log.gz 2024-09-22T01:01:57.049 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56434.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66408.log 2024-09-22T01:01:57.050 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125231.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55701.log 2024-09-22T01:01:57.050 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66408.log: gzip 90.1% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.80062.log.gz --verbose 2024-09-22T01:01:57.051 INFO:teuthology.orchestra.run.smithi060.stderr: -- /var/log/ceph/cluster1-client.admin.73101.log 2024-09-22T01:01:57.051 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.120393.log.gz 2024-09-22T01:01:57.051 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55701.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61698.log 2024-09-22T01:01:57.052 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.73101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111486.log 2024-09-22T01:01:57.052 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80736.log 2024-09-22T01:01:57.053 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.99886.log.gz 2024-09-22T01:01:57.053 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111486.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119860.log.gz 2024-09-22T01:01:57.053 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75547.log 2024-09-22T01:01:57.053 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.130548.log.gz 2024-09-22T01:01:57.054 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80736.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134789.log 2024-09-22T01:01:57.055 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.75547.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56163.log 2024-09-22T01:01:57.055 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.130715.log.gz 2024-09-22T01:01:57.055 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134789.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99467.log 2024-09-22T01:01:57.056 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89638.log.gz 2024-09-22T01:01:57.056 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56163.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89678.log.gz 2024-09-22T01:01:57.056 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78296.log 2024-09-22T01:01:57.056 INFO:teuthology.orchestra.run.smithi060.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-client.admin.64043.log.gz 2024-09-22T01:01:57.057 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99467.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77384.log 2024-09-22T01:01:57.057 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78296.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93418.log 2024-09-22T01:01:57.058 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77384.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107752.log 2024-09-22T01:01:57.058 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93418.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134539.log.gz 2024-09-22T01:01:57.059 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52512.log 2024-09-22T01:01:57.059 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75506.log 2024-09-22T01:01:57.059 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.52512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124449.log 2024-09-22T01:01:57.060 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75506.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120899.log 2024-09-22T01:01:57.061 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124449.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.63674.log.gz 2024-09-22T01:01:57.061 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45432.log 2024-09-22T01:01:57.061 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115979.log.gz 2024-09-22T01:01:57.061 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.120899.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57043.log.gz 2024-09-22T01:01:57.062 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96920.log 2024-09-22T01:01:57.062 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45432.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109334.log 2024-09-22T01:01:57.063 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.96920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66520.log 2024-09-22T01:01:57.063 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.109334.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134621.log 2024-09-22T01:01:57.064 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66520.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102641.log 2024-09-22T01:01:57.064 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.104887.log.gz 2024-09-22T01:01:57.064 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134621.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29516.log 2024-09-22T01:01:57.065 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102641.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84501.log 2024-09-22T01:01:57.065 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.29516.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29516.log.gz 2024-09-22T01:01:57.066 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79178.log 2024-09-22T01:01:57.066 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84501.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78716.log 2024-09-22T01:01:57.067 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79178.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123092.log 2024-09-22T01:01:57.067 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111181.log 2024-09-22T01:01:57.067 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123092.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64069.log 2024-09-22T01:01:57.068 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111181.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95552.log 2024-09-22T01:01:57.069 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.64069.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132804.log 2024-09-22T01:01:57.069 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.95552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105163.log 2024-09-22T01:01:57.069 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.75547.log.gz 2024-09-22T01:01:57.070 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.132804.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54848.log 2024-09-22T01:01:57.070 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72990.log 2024-09-22T01:01:57.071 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54848.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127081.log 2024-09-22T01:01:57.071 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72990.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75805.log 2024-09-22T01:01:57.072 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127081.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140447.log 2024-09-22T01:01:57.073 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.75805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82035.log 2024-09-22T01:01:57.074 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66583.log 2024-09-22T01:01:57.074 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82035.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100541.log.gz 2024-09-22T01:01:57.074 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32497.log 2024-09-22T01:01:57.075 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31287.log 2024-09-22T01:01:57.075 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32497.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50209.log 2024-09-22T01:01:57.075 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31287.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83968.log 2024-09-22T01:01:57.076 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50209.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39703.log 2024-09-22T01:01:57.076 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83968.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.133789.log.gz 2024-09-22T01:01:57.076 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59312.log 2024-09-22T01:01:57.076 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39703.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.40358.log.gz 2024-09-22T01:01:57.077 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105012.log 2024-09-22T01:01:57.078 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67884.log 2024-09-22T01:01:57.078 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105012.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113251.log.gz 2024-09-22T01:01:57.078 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35377.log 2024-09-22T01:01:57.079 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67884.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93690.log 2024-09-22T01:01:57.079 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35377.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80608.log 2024-09-22T01:01:57.080 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93690.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90899.log 2024-09-22T01:01:57.080 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113544.log 2024-09-22T01:01:57.081 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58052.log 2024-09-22T01:01:57.081 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113544.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92159.log.gz 2024-09-22T01:01:57.081 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41498.log 2024-09-22T01:01:57.082 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58052.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118785.log 2024-09-22T01:01:57.082 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41498.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40347.log 2024-09-22T01:01:57.083 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104952.log 2024-09-22T01:01:57.083 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.76671.log.gz 2024-09-22T01:01:57.083 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40347.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127711.log 2024-09-22T01:01:57.084 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104952.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77501.log 2024-09-22T01:01:57.085 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127711.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71125.log 2024-09-22T01:01:57.085 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77501.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67422.log 2024-09-22T01:01:57.086 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71125.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36049.log 2024-09-22T01:01:57.086 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67422.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102912.log 2024-09-22T01:01:57.087 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36049.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44383.log 2024-09-22T01:01:57.087 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.35208.log.gz 2024-09-22T01:01:57.087 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103208.log 2024-09-22T01:01:57.088 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53471.log 2024-09-22T01:01:57.088 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131232.log 2024-09-22T01:01:57.089 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100688.log 2024-09-22T01:01:57.089 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.131232.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43679.log.gz 2024-09-22T01:01:57.089 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73747.log 2024-09-22T01:01:57.090 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100688.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43434.log 2024-09-22T01:01:57.090 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.73747.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74314.log 2024-09-22T01:01:57.091 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43434.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33128.log 2024-09-22T01:01:57.091 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.74314.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74314.log.gz 2024-09-22T01:01:57.091 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32791.log.gz 2024-09-22T01:01:57.091 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138282.log 2024-09-22T01:01:57.092 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44592.log 2024-09-22T01:01:57.093 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138282.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88473.log 2024-09-22T01:01:57.093 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112350.log 2024-09-22T01:01:57.094 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88473.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136722.log 2024-09-22T01:01:57.094 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91528.log 2024-09-22T01:01:57.095 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102284.log 2024-09-22T01:01:57.095 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91528.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59902.log 2024-09-22T01:01:57.096 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102284.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97303.log 2024-09-22T01:01:57.096 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122582.log 2024-09-22T01:01:57.097 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.97303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42416.log 2024-09-22T01:01:57.097 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46274.log 2024-09-22T01:01:57.098 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.42416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58641.log 2024-09-22T01:01:57.098 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46274.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111282.log 2024-09-22T01:01:57.099 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58641.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40860.log 2024-09-22T01:01:57.100 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111282.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.64415.log.gz 2024-09-22T01:01:57.100 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104217.log 2024-09-22T01:01:57.100 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.39173.log.gz 2024-09-22T01:01:57.100 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.40860.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46485.log 2024-09-22T01:01:57.101 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104217.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72993.log 2024-09-22T01:01:57.101 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46485.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54442.log 2024-09-22T01:01:57.102 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79306.log 2024-09-22T01:01:57.103 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45348.log 2024-09-22T01:01:57.104 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55744.log 2024-09-22T01:01:57.104 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66408.log.gz 2024-09-22T01:01:57.104 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45348.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35142.log 2024-09-22T01:01:57.104 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55744.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57634.log 2024-09-22T01:01:57.105 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35142.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82581.log 2024-09-22T01:01:57.106 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57634.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68624.log.gz 2024-09-22T01:01:57.106 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97262.log 2024-09-22T01:01:57.106 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82581.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90833.log 2024-09-22T01:01:57.107 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.97262.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121409.log 2024-09-22T01:01:57.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95209.log 2024-09-22T01:01:57.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.121409.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126805.log 2024-09-22T01:01:57.108 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67003.log.gz 2024-09-22T01:01:57.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.95209.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119009.log 2024-09-22T01:01:57.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142947.log 2024-09-22T01:01:57.110 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.119009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114659.log 2024-09-22T01:01:57.111 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.142947.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142947.log.gz 2024-09-22T01:01:57.111 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102703.log 2024-09-22T01:01:57.111 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.87511.log.gz 2024-09-22T01:01:57.111 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35880.log 2024-09-22T01:01:57.112 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121836.log 2024-09-22T01:01:57.112 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53899.log 2024-09-22T01:01:57.113 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121836.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127246.log 2024-09-22T01:01:57.114 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.53899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87657.log 2024-09-22T01:01:57.114 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127246.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64842.log 2024-09-22T01:01:57.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87657.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91378.log 2024-09-22T01:01:57.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64842.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117781.log 2024-09-22T01:01:57.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91378.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94926.log 2024-09-22T01:01:57.116 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137566.log.gz 2024-09-22T01:01:57.116 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.117781.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112771.log 2024-09-22T01:01:57.117 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.94926.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139775.log 2024-09-22T01:01:57.118 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109868.log 2024-09-22T01:01:57.118 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139775.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99230.log 2024-09-22T01:01:57.119 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58326.log.gz 2024-09-22T01:01:57.119 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104532.log 2024-09-22T01:01:57.119 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99230.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35207.log 2024-09-22T01:01:57.120 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60951.log 2024-09-22T01:01:57.120 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35207.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.82919.log.gz 2024-09-22T01:01:57.121 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46084.log 2024-09-22T01:01:57.121 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.60951.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.60951.log.gz 2024-09-22T01:01:57.121 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34452.log 2024-09-22T01:01:57.122 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46084.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51965.log 2024-09-22T01:01:57.123 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34452.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108696.log 2024-09-22T01:01:57.123 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51965.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.89699.log.gz 2024-09-22T01:01:57.123 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33401.log 2024-09-22T01:01:57.124 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.108696.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98529.log 2024-09-22T01:01:57.124 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33401.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94371.log 2024-09-22T01:01:57.125 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98529.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67842.log 2024-09-22T01:01:57.125 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94371.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80145.log 2024-09-22T01:01:57.126 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74900.log 2024-09-22T01:01:57.126 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80526.log.gz 2024-09-22T01:01:57.127 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80145.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46572.log 2024-09-22T01:01:57.127 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.74900.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74093.log 2024-09-22T01:01:57.127 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46572.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115770.log 2024-09-22T01:01:57.128 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.74093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37729.log 2024-09-22T01:01:57.129 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142409.log 2024-09-22T01:01:57.129 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93630.log.gz 2024-09-22T01:01:57.129 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.91210.log.gz 2024-09-22T01:01:57.129 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37729.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90118.log 2024-09-22T01:01:57.130 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.142409.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.142409.log.gz 2024-09-22T01:01:57.130 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60488.log 2024-09-22T01:01:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90118.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77337.log 2024-09-22T01:01:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60488.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96642.log 2024-09-22T01:01:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77337.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129112.log 2024-09-22T01:01:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71286.log 2024-09-22T01:01:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48714.log 2024-09-22T01:01:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.71286.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111225.log 2024-09-22T01:01:57.134 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48714.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124349.log 2024-09-22T01:01:57.135 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92259.log 2024-09-22T01:01:57.135 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124349.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.99293.log.gz 2024-09-22T01:01:57.135 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83798.log 2024-09-22T01:01:57.136 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92259.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106886.log 2024-09-22T01:01:57.136 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83798.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79935.log 2024-09-22T01:01:57.137 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.106886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92008.log 2024-09-22T01:01:57.138 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79935.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109637.log 2024-09-22T01:01:57.138 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137819.log.gz 2024-09-22T01:01:57.138 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92008.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117877.log 90.3% 2024-09-22T01:01:57.138 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.137818.log.gz 2024-09-22T01:01:57.139 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35712.log 2024-09-22T01:01:57.139 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.117877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138197.log 2024-09-22T01:01:57.140 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78085.log 2024-09-22T01:01:57.140 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123066.log.gz 2024-09-22T01:01:57.141 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138197.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102828.log 2024-09-22T01:01:57.141 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105140.log 2024-09-22T01:01:57.141 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102828.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123131.log.gz 2024-09-22T01:01:57.142 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35668.log 2024-09-22T01:01:57.142 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116339.log 2024-09-22T01:01:57.143 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35668.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53346.log 2024-09-22T01:01:57.143 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116339.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45477.log 2024-09-22T01:01:57.144 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53346.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109741.log 2024-09-22T01:01:57.145 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91379.log 2024-09-22T01:01:57.145 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.109741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57465.log 2024-09-22T01:01:57.145 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57757.log 2024-09-22T01:01:57.146 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57465.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73489.log 2024-09-22T01:01:57.146 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90833.log.gz 2024-09-22T01:01:57.147 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118625.log 2024-09-22T01:01:57.147 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73489.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96867.log 2024-09-22T01:01:57.148 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.118625.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68833.log 2024-09-22T01:01:57.148 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.96867.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115770.log.gz 2024-09-22T01:01:57.148 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48209.log 2024-09-22T01:01:57.149 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82456.log 2024-09-22T01:01:57.149 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48209.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54917.log 2024-09-22T01:01:57.150 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82456.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30822.log 2024-09-22T01:01:57.150 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124449.log.gz 2024-09-22T01:01:57.150 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54917.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100646.log.gz 2024-09-22T01:01:57.151 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52988.log 2024-09-22T01:01:57.151 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.30822.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142295.log 2024-09-22T01:01:57.152 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.52988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83350.log 2024-09-22T01:01:57.152 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.142295.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138238.log 2024-09-22T01:01:57.152 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.142295.log.gz 2024-09-22T01:01:57.153 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.83350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137735.log 2024-09-22T01:01:57.154 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138238.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122582.log.gz 2024-09-22T01:01:57.154 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52116.log 2024-09-22T01:01:57.154 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137735.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114509.log 2024-09-22T01:01:57.154 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.52116.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52116.log.gz 2024-09-22T01:01:57.155 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81490.log 2024-09-22T01:01:57.155 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67885.log 2024-09-22T01:01:57.156 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81490.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106295.log 2024-09-22T01:01:57.156 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67885.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114719.log 2024-09-22T01:01:57.157 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106295.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125861.log.gz 2024-09-22T01:01:57.157 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88280.log 2024-09-22T01:01:57.158 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114719.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87214.log 2024-09-22T01:01:57.158 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88280.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73747.log.gz 2024-09-22T01:01:57.158 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36762.log 2024-09-22T01:01:57.159 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87214.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133781.log 2024-09-22T01:01:57.160 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36762.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139333.log 2024-09-22T01:01:57.160 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.74093.log.gz 2024-09-22T01:01:57.160 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.87657.log.gz 2024-09-22T01:01:57.160 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133781.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68388.log 2024-09-22T01:01:57.161 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.139333.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62464.log 2024-09-22T01:01:57.162 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68388.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41840.log 2024-09-22T01:01:57.162 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.62464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74947.log 2024-09-22T01:01:57.163 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.41840.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132239.log 2024-09-22T01:01:57.163 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.74947.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61087.log 2024-09-22T01:01:57.164 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108126.log 2024-09-22T01:01:57.164 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61087.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.61087.log.gz 2024-09-22T01:01:57.164 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80314.log 2024-09-22T01:01:57.165 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.108126.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51671.log 2024-09-22T01:01:57.166 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80314.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114659.log.gz 2024-09-22T01:01:57.166 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101906.log 2024-09-22T01:01:57.166 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51671.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109256.log 2024-09-22T01:01:57.167 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125127.log 2024-09-22T01:01:57.167 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97779.log 2024-09-22T01:01:57.168 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125127.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50956.log 2024-09-22T01:01:57.168 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.97779.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35710.log 2024-09-22T01:01:57.168 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125231.log.gz 2024-09-22T01:01:57.169 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.50956.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85467.log 2024-09-22T01:01:57.169 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100647.log 2024-09-22T01:01:57.170 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.85467.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.139333.log.gz 2024-09-22T01:01:57.171 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29391.log 2024-09-22T01:01:57.171 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.138282.log.gz 2024-09-22T01:01:57.171 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100647.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61706.log 2024-09-22T01:01:57.171 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.29391.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29391.log.gz 2024-09-22T01:01:57.171 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136722.log.gz 2024-09-22T01:01:57.172 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77284.log 2024-09-22T01:01:57.173 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61706.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42605.log 2024-09-22T01:01:57.173 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77284.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102786.log 2024-09-22T01:01:57.174 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42605.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67674.log 2024-09-22T01:01:57.174 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102786.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100308.log 2024-09-22T01:01:57.175 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67674.log: gzip 90.7% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.91378.log.gz 2024-09-22T01:01:57.175 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.84703.log 2024-09-22T01:01:57.175 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100308.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77383.log 2024-09-22T01:01:57.176 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88530.log 2024-09-22T01:01:57.176 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77383.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29133.log 2024-09-22T01:01:57.177 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88530.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90329.log.gz 2024-09-22T01:01:57.177 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68474.log 2024-09-22T01:01:57.177 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29133.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29133.log.gz 2024-09-22T01:01:57.178 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131775.log 2024-09-22T01:01:57.178 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68474.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128070.log 2024-09-22T01:01:57.179 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.131775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142921.log 2024-09-22T01:01:57.179 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.75506.log.gz 2024-09-22T01:01:57.180 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.128070.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102956.log 2024-09-22T01:01:57.180 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.142921.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142921.log.gz 2024-09-22T01:01:57.180 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.106886.log.gz 2024-09-22T01:01:57.180 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.108126.log.gz 2024-09-22T01:01:57.181 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87999.log 2024-09-22T01:01:57.181 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102956.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45013.log 2024-09-22T01:01:57.182 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87999.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46443.log 2024-09-22T01:01:57.182 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45013.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128011.log.gz 2024-09-22T01:01:57.183 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122965.log 2024-09-22T01:01:57.183 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35669.log 2024-09-22T01:01:57.184 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122965.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40352.log 2024-09-22T01:01:57.184 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35669.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96940.log 2024-09-22T01:01:57.185 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119114.log 2024-09-22T01:01:57.185 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.96940.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127582.log 2024-09-22T01:01:57.186 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.132804.log.gz 2024-09-22T01:01:57.186 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.119114.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119114.log.gz 2024-09-22T01:01:57.186 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36193.log 2024-09-22T01:01:57.187 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139593.log 2024-09-22T01:01:57.187 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62742.log 2024-09-22T01:01:57.188 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139593.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112981.log 2024-09-22T01:01:57.188 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44679.log 2024-09-22T01:01:57.189 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112981.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72583.log 2024-09-22T01:01:57.189 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44679.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.134789.log.gz 2024-09-22T01:01:57.190 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46952.log 2024-09-22T01:01:57.190 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72583.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129578.log 2024-09-22T01:01:57.191 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109111.log 2024-09-22T01:01:57.191 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129578.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135651.log.gz 2024-09-22T01:01:57.191 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63026.log 2024-09-22T01:01:57.192 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.109111.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33612.log 2024-09-22T01:01:57.195 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.63026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138492.log 2024-09-22T01:01:57.195 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33612.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86408.log 2024-09-22T01:01:57.195 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57337.log 2024-09-22T01:01:57.195 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.86408.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79765.log 2024-09-22T01:01:57.195 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57337.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84371.log 2024-09-22T01:01:57.196 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79765.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35272.log 2024-09-22T01:01:57.197 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84371.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126216.log 2024-09-22T01:01:57.197 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136637.log 2024-09-22T01:01:57.197 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102323.log 2024-09-22T01:01:57.197 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134117.log 2024-09-22T01:01:57.198 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115811.log 2024-09-22T01:01:57.198 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134117.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130803.log 2024-09-22T01:01:57.199 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115811.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30040.log 2024-09-22T01:01:57.199 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.130803.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104532.log.gz 2024-09-22T01:01:57.199 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127751.log 2024-09-22T01:01:57.200 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.30040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50108.log 2024-09-22T01:01:57.201 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127751.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88001.log.gz 2024-09-22T01:01:57.201 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34512.log.gz 2024-09-22T01:01:57.202 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34536.log.gz 2024-09-22T01:01:57.202 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34661.log.gz 2024-09-22T01:01:57.202 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.50108.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34450.log.gz 2024-09-22T01:01:57.202 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34619.log.gz 2024-09-22T01:01:57.202 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109509.log 2024-09-22T01:01:57.202 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37181.log 2024-09-22T01:01:57.203 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.34410.log.gz 2024-09-22T01:01:57.203 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134244.log 2024-09-22T01:01:57.203 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37181.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34748.log 2024-09-22T01:01:57.204 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134244.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134034.log 2024-09-22T01:01:57.204 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34748.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111864.log 2024-09-22T01:01:57.205 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134034.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112411.log 2024-09-22T01:01:57.205 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111864.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73289.log 2024-09-22T01:01:57.205 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112411.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94437.log 2024-09-22T01:01:57.206 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92218.log 2024-09-22T01:01:57.207 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116087.log 2024-09-22T01:01:57.207 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52190.log 2024-09-22T01:01:57.208 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116087.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59185.log 2024-09-22T01:01:57.208 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.52190.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34828.log 2024-09-22T01:01:57.209 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43961.log 2024-09-22T01:01:57.209 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34828.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47371.log 2024-09-22T01:01:57.210 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43961.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62743.log 2024-09-22T01:01:57.210 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47371.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89153.log 2024-09-22T01:01:57.211 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132851.log 2024-09-22T01:01:57.211 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89153.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36679.log 2024-09-22T01:01:57.211 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76628.log 2024-09-22T01:01:57.212 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36679.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105853.log 2024-09-22T01:01:57.212 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76628.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44550.log 2024-09-22T01:01:57.213 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.105853.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105853.log.gz 2024-09-22T01:01:57.213 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34451.log 2024-09-22T01:01:57.213 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44550.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.75805.log.gz 2024-09-22T01:01:57.213 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122856.log 2024-09-22T01:01:57.214 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34451.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100444.log 2024-09-22T01:01:57.215 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122856.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33988.log 2024-09-22T01:01:57.215 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100444.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51863.log 2024-09-22T01:01:57.216 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33988.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.34452.log.gz 2024-09-22T01:01:57.216 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99751.log 2024-09-22T01:01:57.217 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51863.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44133.log 2024-09-22T01:01:57.217 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129022.log 2024-09-22T01:01:57.217 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44133.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35142.log.gz 2024-09-22T01:01:57.218 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36343.log 2024-09-22T01:01:57.218 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.129022.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58725.log 2024-09-22T01:01:57.219 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36343.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100392.log 2024-09-22T01:01:57.219 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58725.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134118.log 2024-09-22T01:01:57.220 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100392.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35710.log.gz 2024-09-22T01:01:57.220 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106261.log 2024-09-22T01:01:57.221 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134118.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37857.log 2024-09-22T01:01:57.221 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106261.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.35712.log.gz 2024-09-22T01:01:57.221 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79305.log 2024-09-22T01:01:57.222 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37857.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54442.log.gz 2024-09-22T01:01:57.222 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62054.log 2024-09-22T01:01:57.223 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79305.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45601.log 2024-09-22T01:01:57.223 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62054.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86428.log 2024-09-22T01:01:57.224 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59815.log 2024-09-22T01:01:57.224 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.86428.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52559.log 2024-09-22T01:01:57.224 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59815.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59312.log.gz 2024-09-22T01:01:57.225 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112704.log 2024-09-22T01:01:57.225 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.52559.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112874.log 2024-09-22T01:01:57.226 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112704.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109547.log 2024-09-22T01:01:57.226 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112874.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85993.log 2024-09-22T01:01:57.227 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109547.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33908.log 2024-09-22T01:01:57.227 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.85993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33820.log 2024-09-22T01:01:57.228 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136429.log 2024-09-22T01:01:57.229 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33820.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.38635.log.gz 2024-09-22T01:01:57.229 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133648.log 2024-09-22T01:01:57.229 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.38693.log.gz 2024-09-22T01:01:57.229 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33698.log 2024-09-22T01:01:57.230 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133648.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80407.log 2024-09-22T01:01:57.230 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122815.log 2024-09-22T01:01:57.231 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80407.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69672.log 2024-09-22T01:01:57.231 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122815.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100897.log 2024-09-22T01:01:57.231 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69672.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98364.log 2024-09-22T01:01:57.232 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100897.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141436.log 2024-09-22T01:01:57.232 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98364.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.39703.log.gz 2024-09-22T01:01:57.233 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129276.log 2024-09-22T01:01:57.234 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.141436.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141436.log.gz 2024-09-22T01:01:57.234 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54639.log 2024-09-22T01:01:57.234 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129276.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33949.log 2024-09-22T01:01:57.235 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.54639.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58261.log 2024-09-22T01:01:57.235 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84021.log 2024-09-22T01:01:57.236 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58261.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52585.log 2024-09-22T01:01:57.236 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.84021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56750.log 2024-09-22T01:01:57.237 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52585.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140863.log 2024-09-22T01:01:57.237 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56750.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123552.log 2024-09-22T01:01:57.238 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45601.log.gz 2024-09-22T01:01:57.238 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.140863.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131945.log 2024-09-22T01:01:57.238 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112726.log 2024-09-22T01:01:57.239 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72421.log 2024-09-22T01:01:57.240 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112726.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45180.log 2024-09-22T01:01:57.240 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72421.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125063.log 2024-09-22T01:01:57.240 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46952.log.gz 2024-09-22T01:01:57.241 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45180.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59561.log 2024-09-22T01:01:57.241 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125063.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67253.log 2024-09-22T01:01:57.242 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132748.log 2024-09-22T01:01:57.242 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67253.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89739.log 2024-09-22T01:01:57.243 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132748.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57841.log 2024-09-22T01:01:57.244 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127332.log 2024-09-22T01:01:57.244 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57841.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43112.log 2024-09-22T01:01:57.245 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127332.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29466.log 2024-09-22T01:01:57.245 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91839.log 2024-09-22T01:01:57.246 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29466.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29466.log.gz 2024-09-22T01:01:57.246 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103752.log 2024-09-22T01:01:57.246 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53471.log.gz 2024-09-22T01:01:57.246 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91839.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82416.log 2024-09-22T01:01:57.247 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118464.log 2024-09-22T01:01:57.248 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54873.log 2024-09-22T01:01:57.248 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.118464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110144.log 2024-09-22T01:01:57.249 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54873.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57592.log 2024-09-22T01:01:57.249 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110144.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101737.log 2024-09-22T01:01:57.250 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115143.log 2024-09-22T01:01:57.250 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67422.log.gz 2024-09-22T01:01:57.250 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64374.log.gz 2024-09-22T01:01:57.251 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101737.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133524.log 2024-09-22T01:01:57.251 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69986.log 2024-09-22T01:01:57.252 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133524.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.65095.log.gz 2024-09-22T01:01:57.252 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46212.log 2024-09-22T01:01:57.252 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69986.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29315.log 2024-09-22T01:01:57.253 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46212.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-09-22T01:01:57.253 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29315.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29315.log.gz 2024-09-22T01:01:57.253 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114930.log 2024-09-22T01:01:57.254 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124536.log 2024-09-22T01:01:57.255 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114930.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62464.log.gz 2024-09-22T01:01:57.255 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78613.log 2024-09-22T01:01:57.255 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90120.log 2024-09-22T01:01:57.256 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78613.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85317.log 2024-09-22T01:01:57.256 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.90120.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102640.log 2024-09-22T01:01:57.257 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.85317.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85317.log.gz 2024-09-22T01:01:57.257 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32332.log 2024-09-22T01:01:57.258 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104467.log 2024-09-22T01:01:57.258 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43635.log 2024-09-22T01:01:57.259 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42605.log.gz 2024-09-22T01:01:57.259 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104467.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29938.log 2024-09-22T01:01:57.259 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29904.log 2024-09-22T01:01:57.260 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.29938.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36722.log 2024-09-22T01:01:57.260 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.29904.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29741.log 2024-09-22T01:01:57.261 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116514.log 2024-09-22T01:01:57.261 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29741.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29741.log.gz 2024-09-22T01:01:57.262 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141524.log 2024-09-22T01:01:57.262 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69773.log 2024-09-22T01:01:57.263 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.141524.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56919.log 2024-09-22T01:01:57.263 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69773.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88280.log.gz 2024-09-22T01:01:57.264 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142687.log 2024-09-22T01:01:57.264 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56919.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107621.log 2024-09-22T01:01:57.265 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91748.log.gz 2024-09-22T01:01:57.265 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.142687.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142687.log.gz 2024-09-22T01:01:57.265 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.141524.log.gz 2024-09-22T01:01:57.265 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32878.log 2024-09-22T01:01:57.265 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91839.log.gz 2024-09-22T01:01:57.266 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107621.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.91379.log.gz 2024-09-22T01:01:57.266 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35271.log 2024-09-22T01:01:57.267 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32878.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47413.log 2024-09-22T01:01:57.268 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139397.log 2024-09-22T01:01:57.268 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.47413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64673.log 2024-09-22T01:01:57.268 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.139397.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94699.log 2024-09-22T01:01:57.269 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.64673.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48503.log 2024-09-22T01:01:57.269 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.82416.log.gz 2024-09-22T01:01:57.270 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94699.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97148.log 2024-09-22T01:01:57.270 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.82456.log.gz 2024-09-22T01:01:57.271 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141192.log 2024-09-22T01:01:57.271 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.97148.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59652.log 2024-09-22T01:01:57.271 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98364.log.gz 2024-09-22T01:01:57.272 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.141192.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135295.log 2024-09-22T01:01:57.272 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60698.log 2024-09-22T01:01:57.273 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135295.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62269.log 2024-09-22T01:01:57.273 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69043.log 2024-09-22T01:01:57.274 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128980.log 2024-09-22T01:01:57.274 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66495.log 2024-09-22T01:01:57.275 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.128980.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66732.log 2024-09-22T01:01:57.275 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66495.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97008.log 2024-09-22T01:01:57.276 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66732.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99751.log.gz 2024-09-22T01:01:57.276 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126701.log 2024-09-22T01:01:57.276 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.97008.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100647.log.gz 2024-09-22T01:01:57.276 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56707.log 2024-09-22T01:01:57.277 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126701.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41685.log 2024-09-22T01:01:57.278 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56707.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51121.log 2024-09-22T01:01:57.279 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41685.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28879.log 2024-09-22T01:01:57.279 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.51121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60741.log 2024-09-22T01:01:57.280 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.28879.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39074.log 2024-09-22T01:01:57.280 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100444.log.gz 2024-09-22T01:01:57.280 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102788.log 2024-09-22T01:01:57.281 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.39074.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98676.log 2024-09-22T01:01:57.282 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102788.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40785.log 2024-09-22T01:01:57.282 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78591.log 2024-09-22T01:01:57.283 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.40785.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40785.log.gz 2024-09-22T01:01:57.283 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61743.log 2024-09-22T01:01:57.284 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78591.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126639.log 2024-09-22T01:01:57.284 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93520.log 2024-09-22T01:01:57.284 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126639.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137734.log 2024-09-22T01:01:57.285 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58177.log.gz 2024-09-22T01:01:57.285 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93520.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76037.log 2024-09-22T01:01:57.286 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137734.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112726.log.gz 2024-09-22T01:01:57.286 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45558.log 2024-09-22T01:01:57.287 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76037.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65391.log 2024-09-22T01:01:57.287 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45558.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124854.log 2024-09-22T01:01:57.288 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.65391.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132387.log 2024-09-22T01:01:57.288 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124854.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99468.log 2024-09-22T01:01:57.289 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132387.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132029.log 2024-09-22T01:01:57.289 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99468.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39615.log 2024-09-22T01:01:57.290 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.132029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134581.log 2024-09-22T01:01:57.290 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39615.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113459.log 2024-09-22T01:01:57.291 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134581.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70992.log 2024-09-22T01:01:57.291 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.113459.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129757.log 2024-09-22T01:01:57.292 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35269.log 2024-09-22T01:01:57.293 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44382.log 2024-09-22T01:01:57.293 INFO:teuthology.orchestra.run.smithi060.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.131326.log.gz 2024-09-22T01:01:57.293 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54405.log 2024-09-22T01:01:57.293 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126176.log 2024-09-22T01:01:57.294 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54405.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91443.log 2024-09-22T01:01:57.295 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126176.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123447.log 2024-09-22T01:01:57.295 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123802.log 2024-09-22T01:01:57.295 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139251.log 2024-09-22T01:01:57.296 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123299.log 2024-09-22T01:01:57.297 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.139251.log: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132388.log.gz 2024-09-22T01:01:57.297 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50563.log 2024-09-22T01:01:57.297 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82373.log 2024-09-22T01:01:57.298 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50563.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116131.log 2024-09-22T01:01:57.298 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82373.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29541.log 2024-09-22T01:01:57.299 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137355.log 2024-09-22T01:01:57.299 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.29541.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29541.log.gz 2024-09-22T01:01:57.299 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96794.log 2024-09-22T01:01:57.300 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137355.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34788.log 2024-09-22T01:01:57.300 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96794.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96794.log.gz 2024-09-22T01:01:57.300 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29141.log 2024-09-22T01:01:57.301 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34788.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42199.log 2024-09-22T01:01:57.301 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.29141.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29141.log.gz 2024-09-22T01:01:57.302 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54916.log 2024-09-22T01:01:57.302 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42199.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.94371.log.gz 2024-09-22T01:01:57.303 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38916.log 2024-09-22T01:01:57.303 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54916.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34577.log 2024-09-22T01:01:57.304 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30074.log 2024-09-22T01:01:57.305 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34577.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31323.log 2024-09-22T01:01:57.305 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.30074.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30074.log.gz 2024-09-22T01:01:57.305 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131031.log 2024-09-22T01:01:57.305 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.118625.log.gz 2024-09-22T01:01:57.306 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83958.log 2024-09-22T01:01:57.306 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.131031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103628.log 2024-09-22T01:01:57.307 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83958.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43197.log 2024-09-22T01:01:57.308 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103628.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48587.log 2024-09-22T01:01:57.308 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43197.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42668.log 2024-09-22T01:01:57.308 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48587.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101232.log 2024-09-22T01:01:57.309 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42668.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121340.log 2024-09-22T01:01:57.309 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101232.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72288.log 2024-09-22T01:01:57.310 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78926.log 2024-09-22T01:01:57.310 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.48209.log.gz 2024-09-22T01:01:57.311 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.72288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92638.log 2024-09-22T01:01:57.311 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78926.log: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.99083.log.gz 2024-09-22T01:01:57.312 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54193.log 2024-09-22T01:01:57.312 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.99119.log.gz 2024-09-22T01:01:57.312 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92638.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38258.log 2024-09-22T01:01:57.313 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81531.log 2024-09-22T01:01:57.313 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.38258.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38258.log.gz 2024-09-22T01:01:57.313 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136894.log 2024-09-22T01:01:57.314 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81531.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80818.log 2024-09-22T01:01:57.315 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132747.log 2024-09-22T01:01:57.315 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80818.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53921.log 2024-09-22T01:01:57.315 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132747.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136301.log 2024-09-22T01:01:57.316 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53921.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109511.log 2024-09-22T01:01:57.317 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.136301.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88330.log 2024-09-22T01:01:57.317 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109511.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85343.log 2024-09-22T01:01:57.318 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88330.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32270.log 2024-09-22T01:01:57.318 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48128.log.gz 2024-09-22T01:01:57.318 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.85343.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85343.log.gz 2024-09-22T01:01:57.318 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52104.log 2024-09-22T01:01:57.319 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.32270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87972.log 2024-09-22T01:01:57.320 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.52104.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52104.log.gz 2024-09-22T01:01:57.320 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72454.log 2024-09-22T01:01:57.320 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79474.log 2024-09-22T01:01:57.321 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116963.log.gz 2024-09-22T01:01:57.321 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116966.log.gz 2024-09-22T01:01:57.321 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121219.log 2024-09-22T01:01:57.321 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.133781.log.gz 2024-09-22T01:01:57.322 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79474.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110696.log 2024-09-22T01:01:57.322 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126953.log 2024-09-22T01:01:57.323 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110696.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140686.log 2024-09-22T01:01:57.323 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126953.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45181.log 2024-09-22T01:01:57.324 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.140686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124030.log 2024-09-22T01:01:57.324 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45181.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135922.log 2024-09-22T01:01:57.325 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106491.log 2024-09-22T01:01:57.325 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135922.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42200.log 2024-09-22T01:01:57.326 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106491.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122882.log 2024-09-22T01:01:57.326 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55492.log.gz 2024-09-22T01:01:57.327 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.42200.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118387.log 2024-09-22T01:01:57.327 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33275.log 2024-09-22T01:01:57.328 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.118387.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30245.log 2024-09-22T01:01:57.328 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45770.log 2024-09-22T01:01:57.329 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.30245.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89404.log 2024-09-22T01:01:57.329 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.108696.log.gz 2024-09-22T01:01:57.329 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67154.log 2024-09-22T01:01:57.330 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89404.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48462.log 2024-09-22T01:01:57.330 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67154.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64157.log 2024-09-22T01:01:57.331 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.48462.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70657.log 2024-09-22T01:01:57.332 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.64157.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69460.log 2024-09-22T01:01:57.332 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70657.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109091.log 2024-09-22T01:01:57.333 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80358.log 2024-09-22T01:01:57.333 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.109091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70303.log 2024-09-22T01:01:57.334 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80358.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.74920.log.gz 2024-09-22T01:01:57.334 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60151.log 2024-09-22T01:01:57.334 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.131775.log.gz 2024-09-22T01:01:57.334 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139647.log 2024-09-22T01:01:57.335 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60151.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80650.log 2024-09-22T01:01:57.336 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139647.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37519.log 2024-09-22T01:01:57.336 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80650.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75673.log 2024-09-22T01:01:57.337 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133179.log 2024-09-22T01:01:57.337 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.75673.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122309.log 2024-09-22T01:01:57.338 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133179.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89068.log 2024-09-22T01:01:57.338 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89992.log 2024-09-22T01:01:57.339 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83796.log 2024-09-22T01:01:57.339 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80946.log 2024-09-22T01:01:57.340 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83796.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105827.log 2024-09-22T01:01:57.340 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80946.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108293.log 2024-09-22T01:01:57.341 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.105827.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105827.log.gz 2024-09-22T01:01:57.341 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139397.log.gz 2024-09-22T01:01:57.341 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73197.log 2024-09-22T01:01:57.342 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55053.log.gz 2024-09-22T01:01:57.342 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.108293.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94437.log.gz 2024-09-22T01:01:57.342 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51257.log 2024-09-22T01:01:57.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73197.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126869.log 2024-09-22T01:01:57.343 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.51257.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135967.log 2024-09-22T01:01:57.344 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126869.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116129.log 2024-09-22T01:01:57.345 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135967.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86496.log 2024-09-22T01:01:57.345 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116129.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.107124.log.gz 2024-09-22T01:01:57.345 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108267.log 2024-09-22T01:01:57.346 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.86496.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110396.log 2024-09-22T01:01:57.346 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.108267.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92050.log 2024-09-22T01:01:57.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64338.log 2024-09-22T01:01:57.347 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.92050.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43476.log 2024-09-22T01:01:57.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64338.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140482.log 2024-09-22T01:01:57.348 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.43476.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140756.log 2024-09-22T01:01:57.349 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109511.log.gz 2024-09-22T01:01:57.349 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.140482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105099.log 2024-09-22T01:01:57.350 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121664.log 2024-09-22T01:01:57.350 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105099.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51964.log 2024-09-22T01:01:57.351 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121664.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91211.log 2024-09-22T01:01:57.351 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.51964.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115980.log 2024-09-22T01:01:57.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91211.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108220.log 2024-09-22T01:01:57.352 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115980.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.73197.log.gz 2024-09-22T01:01:57.352 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.120899.log.gz 2024-09-22T01:01:57.352 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99694.log 2024-09-22T01:01:57.353 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.108220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33211.log 2024-09-22T01:01:57.354 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99694.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67370.log 2024-09-22T01:01:57.354 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33211.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103248.log 2024-09-22T01:01:57.355 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67370.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119151.log 2024-09-22T01:01:57.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103248.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38086.log 2024-09-22T01:01:57.356 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.119151.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119151.log.gz 2024-09-22T01:01:57.356 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.108293.log.gz 2024-09-22T01:01:57.356 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104112.log 2024-09-22T01:01:57.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.38086.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89573.log 2024-09-22T01:01:57.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57001.log 2024-09-22T01:01:57.358 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89573.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68201.log 2024-09-22T01:01:57.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100540.log 2024-09-22T01:01:57.359 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79977.log 2024-09-22T01:01:57.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94151.log 2024-09-22T01:01:57.360 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79977.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82689.log 2024-09-22T01:01:57.361 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94151.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79093.log 2024-09-22T01:01:57.361 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82689.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89572.log 2024-09-22T01:01:57.361 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110144.log.gz 2024-09-22T01:01:57.361 INFO:teuthology.orchestra.run.smithi060.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51121.log.gz 2024-09-22T01:01:57.362 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65664.log 2024-09-22T01:01:57.363 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.89572.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132682.log 2024-09-22T01:01:57.363 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.65664.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135167.log 2024-09-22T01:01:57.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132682.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62645.log 2024-09-22T01:01:57.364 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.135167.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75242.log 2024-09-22T01:01:57.365 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.62645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91503.log 2024-09-22T01:01:57.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75242.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44971.log 2024-09-22T01:01:57.366 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91503.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.106261.log.gz 2024-09-22T01:01:57.366 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77185.log 2024-09-22T01:01:57.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75356.log 2024-09-22T01:01:57.367 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51184.log 2024-09-22T01:01:57.368 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75356.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131440.log 2024-09-22T01:01:57.368 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.51184.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60532.log 2024-09-22T01:01:57.369 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131440.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115813.log 2024-09-22T01:01:57.369 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112307.log 2024-09-22T01:01:57.370 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.115813.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55658.log 2024-09-22T01:01:57.370 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59355.log 2024-09-22T01:01:57.371 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140698.log.gz 2024-09-22T01:01:57.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55658.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73866.log 2024-09-22T01:01:57.371 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59355.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83509.log 2024-09-22T01:01:57.372 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.73866.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94502.log 2024-09-22T01:01:57.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101106.log 2024-09-22T01:01:57.373 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.94502.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94502.log.gz 2024-09-22T01:01:57.373 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131476.log 2024-09-22T01:01:57.374 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101106.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78338.log 2024-09-22T01:01:57.374 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.131476.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62419.log 2024-09-22T01:01:57.375 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78338.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139251.log.gz 2024-09-22T01:01:57.375 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122389.log 2024-09-22T01:01:57.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.62419.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38085.log 2024-09-22T01:01:57.376 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88279.log 2024-09-22T01:01:57.377 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.38085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78548.log 2024-09-22T01:01:57.377 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88279.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107251.log 2024-09-22T01:01:57.378 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.78548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45056.log 2024-09-22T01:01:57.378 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.107251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70514.log 2024-09-22T01:01:57.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111990.log 2024-09-22T01:01:57.379 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.70514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139310.log 2024-09-22T01:01:57.380 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.111990.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102158.log 2024-09-22T01:01:57.380 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114930.log.gz 2024-09-22T01:01:57.381 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.139310.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.139310.log.gz 2024-09-22T01:01:57.381 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129441.log 2024-09-22T01:01:57.381 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52585.log.gz 2024-09-22T01:01:57.382 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102158.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63332.log 2024-09-22T01:01:57.382 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.129441.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124286.log 2024-09-22T01:01:57.383 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.63332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95611.log 2024-09-22T01:01:57.384 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124286.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85440.log 2024-09-22T01:01:57.384 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.33820.log.gz 2024-09-22T01:01:57.384 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.95611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134412.log 2024-09-22T01:01:57.385 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.85440.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91042.log 2024-09-22T01:01:57.385 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134412.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94328.log 2024-09-22T01:01:57.386 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91042.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83837.log 2024-09-22T01:01:57.386 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.94328.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94328.log.gz 2024-09-22T01:01:57.386 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98365.log 2024-09-22T01:01:57.387 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83837.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125306.log 2024-09-22T01:01:57.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.98365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46613.log 2024-09-22T01:01:57.388 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76991.log 2024-09-22T01:01:57.389 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46613.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31729.log 2024-09-22T01:01:57.389 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76991.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140454.log 2024-09-22T01:01:57.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31729.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53310.log 2024-09-22T01:01:57.390 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.140454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59816.log 2024-09-22T01:01:57.391 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53310.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33044.log 2024-09-22T01:01:57.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59816.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133823.log 2024-09-22T01:01:57.392 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.33044.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125127.log.gz 2024-09-22T01:01:57.392 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122438.log 2024-09-22T01:01:57.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133823.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101443.log 2024-09-22T01:01:57.393 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122438.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76969.log 2024-09-22T01:01:57.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68557.log 2024-09-22T01:01:57.394 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.76969.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124854.log.gz 2024-09-22T01:01:57.394 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44678.log 2024-09-22T01:01:57.395 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68557.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50564.log 2024-09-22T01:01:57.396 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.44678.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97645.log 2024-09-22T01:01:57.396 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31243.log 2024-09-22T01:01:57.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.97645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64947.log 2024-09-22T01:01:57.397 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.31243.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127751.log.gz 2024-09-22T01:01:57.397 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83264.log 2024-09-22T01:01:57.399 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.64947.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138702.log 2024-09-22T01:01:57.399 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83264.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.83264.log.gz 2024-09-22T01:01:57.399 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85355.log 2024-09-22T01:01:57.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.138702.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126639.log.gz 2024-09-22T01:01:57.400 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64224.log 2024-09-22T01:01:57.400 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.85355.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85355.log.gz 2024-09-22T01:01:57.400 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60321.log 2024-09-22T01:01:57.401 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64224.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100266.log 2024-09-22T01:01:57.402 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.60321.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53041.log 2024-09-22T01:01:57.402 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100266.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.38086.log.gz 2024-09-22T01:01:57.402 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37349.log 2024-09-22T01:01:57.403 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.53041.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54635.log 2024-09-22T01:01:57.403 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37349.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132007.log 2024-09-22T01:01:57.404 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34158.log 2024-09-22T01:01:57.405 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.132007.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.35269.log.gz 2024-09-22T01:01:57.405 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69816.log 2024-09-22T01:01:57.405 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34158.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127858.log 2024-09-22T01:01:57.406 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69816.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125190.log 2024-09-22T01:01:57.407 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.127858.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.66495.log.gz 2024-09-22T01:01:57.407 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124916.log 2024-09-22T01:01:57.407 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.125190.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41933.log 2024-09-22T01:01:57.408 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.124916.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46741.log 2024-09-22T01:01:57.408 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41933.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94028.log 2024-09-22T01:01:57.409 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68053.log 2024-09-22T01:01:57.409 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.94028.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34032.log 2024-09-22T01:01:57.410 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.68053.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75910.log 2024-09-22T01:01:57.411 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.34032.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134747.log 2024-09-22T01:01:57.411 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67968.log.gz 2024-09-22T01:01:57.411 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.75910.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117159.log 2024-09-22T01:01:57.411 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.134747.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-09-22T01:01:57.412 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.117159.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140537.log 2024-09-22T01:01:57.413 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69187.log 2024-09-22T01:01:57.413 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119175.log 2024-09-22T01:01:57.414 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.69187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64906.log 2024-09-22T01:01:57.414 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70514.log.gz 2024-09-22T01:01:57.414 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.119175.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119175.log.gz 2024-09-22T01:01:57.414 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79853.log 2024-09-22T01:01:57.415 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36469.log 2024-09-22T01:01:57.416 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79853.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84493.log 2024-09-22T01:01:57.416 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.36469.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100016.log 2024-09-22T01:01:57.417 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.84493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99336.log 2024-09-22T01:01:57.418 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100016.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96479.log 2024-09-22T01:01:57.418 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.99336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109446.log 2024-09-22T01:01:57.419 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95060.log 2024-09-22T01:01:57.419 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109446.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109841.log 2024-09-22T01:01:57.420 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.95060.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66163.log 2024-09-22T01:01:57.421 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.109841.log: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.31287.log.gz 2024-09-22T01:01:57.421 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79597.log 2024-09-22T01:01:57.421 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77186.log 2024-09-22T01:01:57.422 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.79597.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129262.log 2024-09-22T01:01:57.423 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77186.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137230.log 2024-09-22T01:01:57.423 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.129262.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91738.log 2024-09-22T01:01:57.424 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32332.log.gz 2024-09-22T01:01:57.424 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.137230.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96325.log 2024-09-22T01:01:57.424 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61582.log 2024-09-22T01:01:57.425 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49679.log 2024-09-22T01:01:57.426 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.61582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88605.log 2024-09-22T01:01:57.426 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.49679.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45770.log.gz 2024-09-22T01:01:57.426 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133137.log 2024-09-22T01:01:57.427 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.88605.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66708.log 2024-09-22T01:01:57.427 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133137.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103122.log 2024-09-22T01:01:57.428 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.66708.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57085.log 2024-09-22T01:01:57.428 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110228.log 2024-09-22T01:01:57.429 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57085.log: 90.1% 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.33949.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.33988.log.gz 2024-09-22T01:01:57.429 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:01:57.429 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101486.log 2024-09-22T01:01:57.430 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45936.log 2024-09-22T01:01:57.431 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.101486.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93562.log 2024-09-22T01:01:57.431 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.45936.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105899.log 2024-09-22T01:01:57.432 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.93562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72050.log 2024-09-22T01:01:57.432 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.105899.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39866.log 2024-09-22T01:01:57.433 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.72050.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64462.log 2024-09-22T01:01:57.433 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.39866.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104886.log 2024-09-22T01:01:57.434 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.64462.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55993.log 2024-09-22T01:01:57.435 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.104886.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91503.log.gz 2024-09-22T01:01:57.435 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116873.log 2024-09-22T01:01:57.435 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.55993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121385.log 2024-09-22T01:01:57.435 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.116873.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.116873.log.gz 2024-09-22T01:01:57.436 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80859.log 2024-09-22T01:01:57.437 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.121385.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112284.log 2024-09-22T01:01:57.437 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.80859.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57171.log 2024-09-22T01:01:57.438 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.112284.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67507.log 2024-09-22T01:01:57.438 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.57171.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58513.log 2024-09-22T01:01:57.439 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67507.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140487.log 2024-09-22T01:01:57.439 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80358.log.gz 2024-09-22T01:01:57.439 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.58513.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140364.log 2024-09-22T01:01:57.440 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91109.log 2024-09-22T01:01:57.441 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.140364.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31054.log 2024-09-22T01:01:57.441 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.91109.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87972.log.gz 2024-09-22T01:01:57.441 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83621.log 2024-09-22T01:01:57.442 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.31054.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103458.log 2024-09-22T01:01:57.443 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.83621.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110950.log 2024-09-22T01:01:57.443 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.103458.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73647.log 2024-09-22T01:01:57.444 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.110950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130224.log 2024-09-22T01:01:57.444 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80946.log.gz 2024-09-22T01:01:57.445 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.73647.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122712.log 2024-09-22T01:01:57.445 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.130224.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130224.log.gz 2024-09-22T01:01:57.445 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116793.log 2024-09-22T01:01:57.446 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.122712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123322.log 2024-09-22T01:01:57.446 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.116793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37142.log 2024-09-22T01:01:57.447 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82799.log 2024-09-22T01:01:57.448 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.37142.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123761.log 2024-09-22T01:01:57.448 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44550.log.gz 2024-09-22T01:01:57.448 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.82799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73038.log 2024-09-22T01:01:57.449 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123761.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52243.log 2024-09-22T01:01:57.450 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.73038.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99968.log 2024-09-22T01:01:57.450 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.52243.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101443.log.gz 2024-09-22T01:01:57.450 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50374.log 2024-09-22T01:01:57.450 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.99968.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.30665.log.gz 2024-09-22T01:01:57.451 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101486.log.gz 2024-09-22T01:01:57.451 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77793.log 2024-09-22T01:01:57.452 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.50374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54319.log 2024-09-22T01:01:57.452 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81029.log 2024-09-22T01:01:57.453 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54319.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.120861.log.gz 2024-09-22T01:01:57.453 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.128246.log.gz -5 2024-09-22T01:01:57.453 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.106401.log 2024-09-22T01:01:57.453 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.81029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67153.log 2024-09-22T01:01:57.454 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.106401.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138702.log.gz 2024-09-22T01:01:57.454 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141611.log 2024-09-22T01:01:57.455 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.67153.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50109.log 2024-09-22T01:01:57.455 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.141611.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.141611.log.gz 2024-09-22T01:01:57.455 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102367.log 2024-09-22T01:01:57.455 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135922.log.gz 2024-09-22T01:01:57.456 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.50109.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63955.log 2024-09-22T01:01:57.456 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102367.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135967.log.gz 2024-09-22T01:01:57.457 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41249.log 2024-09-22T01:01:57.457 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.83837.log.gz 2024-09-22T01:01:57.458 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.63955.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.131476.log.gz 2024-09-22T01:01:57.458 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102116.log 2024-09-22T01:01:57.459 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.41249.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59061.log 2024-09-22T01:01:57.459 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.102116.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112307.log.gz 2024-09-22T01:01:57.459 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54647.log 2024-09-22T01:01:57.460 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.59061.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.99336.log.gz 2024-09-22T01:01:57.460 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53814.log 2024-09-22T01:01:57.460 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.128529.log.gz 2024-09-22T01:01:57.461 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.54647.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56625.log 2024-09-22T01:01:57.462 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.53814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87279.log 2024-09-22T01:01:57.462 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56625.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.89992.log.gz 2024-09-22T01:01:57.462 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130621.log 2024-09-22T01:01:57.463 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87279.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.61582.log.gz 2024-09-22T01:01:57.463 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133907.log 2024-09-22T01:01:57.463 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.mirror.130621.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87547.log 2024-09-22T01:01:57.464 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.133907.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126451.log 2024-09-22T01:01:57.464 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102956.log.gz 2024-09-22T01:01:57.465 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.87547.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35440.log 2024-09-22T01:01:57.465 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.126451.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97236.log 2024-09-22T01:01:57.466 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.35440.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46487.log 2024-09-22T01:01:57.466 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.97236.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72239.log 2024-09-22T01:01:57.467 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.46487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114551.log 2024-09-22T01:01:57.467 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.72239.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.72239.log.gz 2024-09-22T01:01:57.467 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57337.log.gz 2024-09-22T01:01:57.468 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56245.log 2024-09-22T01:01:57.469 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.114551.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100814.log 2024-09-22T01:01:57.469 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.56245.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77833.log 2024-09-22T01:01:57.470 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.100814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123027.log 2024-09-22T01:01:57.471 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.77833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96782.log 2024-09-22T01:01:57.471 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.mirror.123027.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.133524.log.gz 2024-09-22T01:01:57.471 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75061.log 2024-09-22T01:01:57.471 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster2-client.admin.96782.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96782.log.gz 2024-09-22T01:01:57.481 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cluster1-client.admin.75061.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57883.log.gz 2024-09-22T01:01:57.486 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.50108.log.gz 2024-09-22T01:01:57.492 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54917.log.gz 2024-09-22T01:01:57.492 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55744.log.gz 2024-09-22T01:01:57.496 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.50109.log.gz 2024-09-22T01:01:57.504 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57675.log.gz 2024-09-22T01:01:57.508 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54647.log.gz 2024-09-22T01:01:57.518 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54635.log.gz 2024-09-22T01:01:57.519 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63955.log.gz 2024-09-22T01:01:57.520 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57592.log.gz 2024-09-22T01:01:57.524 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57718.log.gz 2024-09-22T01:01:57.529 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56163.log.gz 2024-09-22T01:01:57.533 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56919.log.gz 2024-09-22T01:01:57.540 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56077.log.gz 2024-09-22T01:01:57.541 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56876.log.gz 2024-09-22T01:01:57.570 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.121385.log.gz 2024-09-22T01:01:58.725 INFO:teuthology.orchestra.run.smithi031.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-09-22T01:02:01.440 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 89.8% 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56036.log.gz 2024-09-22T01:02:01.440 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48924.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.56792.log.gz 90.5% 89.9% 2024-09-22T01:02:01.440 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:01.440 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.55318.log.gz 2024-09-22T01:02:01.440 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.48294.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.54108.log.gz 2024-09-22T01:02:01.440 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.588 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.31140.log.gz 2024-09-22T01:02:06.588 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.61677.log.gz 2024-09-22T01:02:06.632 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132682.log.gz 2024-09-22T01:02:06.830 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.86180.log.gz 2024-09-22T01:02:06.831 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.62053.log.gz 2024-09-22T01:02:06.831 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.87399.log.gz 2024-09-22T01:02:06.831 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102996.log.gz 2024-09-22T01:02:06.831 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111508.log.gz 2024-09-22T01:02:06.831 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67507.log.gz 2024-09-22T01:02:06.831 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55183.log.gz 2024-09-22T01:02:06.831 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73398.log.gz 2024-09-22T01:02:06.831 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.34304.log.gz 2024-09-22T01:02:06.831 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81490.log.gz 2024-09-22T01:02:06.831 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.79853.log.gz 2024-09-22T01:02:06.831 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.106688.log.gz 2024-09-22T01:02:06.831 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.31323.log.gz 2024-09-22T01:02:06.831 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31138.log.gz 2024-09-22T01:02:06.832 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.33779.log.gz 2024-09-22T01:02:06.832 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.76880.log.gz 2024-09-22T01:02:06.832 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58725.log.gz 2024-09-22T01:02:06.832 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.124391.log.gz 2024-09-22T01:02:06.832 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33487.log.gz 2024-09-22T01:02:06.832 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.36679.log.gz 2024-09-22T01:02:06.832 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.106637.log.gz 2024-09-22T01:02:06.832 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.76944.log.gz 2024-09-22T01:02:06.832 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.129549.log.gz 2024-09-22T01:02:06.832 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.98787.log.gz 2024-09-22T01:02:06.832 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88041.log.gz 2024-09-22T01:02:06.832 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53557.log.gz 2024-09-22T01:02:06.832 INFO:teuthology.orchestra.run.smithi060.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75356.log.gz 2024-09-22T01:02:06.832 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.72454.log.gz 2024-09-22T01:02:06.832 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.107256.log.gz 2024-09-22T01:02:06.832 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65434.log.gz 2024-09-22T01:02:06.833 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.81415.log.gz 2024-09-22T01:02:06.833 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103458.log.gz 2024-09-22T01:02:06.833 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121218.log.gz 2024-09-22T01:02:06.833 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.31836.log.gz 2024-09-22T01:02:06.833 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90750.log.gz 2024-09-22T01:02:06.833 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30602.log.gz 2024-09-22T01:02:06.833 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.41773.log.gz 2024-09-22T01:02:06.833 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.132599.log.gz 2024-09-22T01:02:06.833 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.116402.log.gz 2024-09-22T01:02:06.833 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137863.log.gz 2024-09-22T01:02:06.833 INFO:teuthology.orchestra.run.smithi060.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.108790.log.gz 2024-09-22T01:02:06.833 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.31351.log.gz 2024-09-22T01:02:06.833 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.81699.log.gz 2024-09-22T01:02:06.833 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46487.log.gz 2024-09-22T01:02:06.833 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.129573.log.gz 2024-09-22T01:02:06.833 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52243.log.gz 2024-09-22T01:02:06.833 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.129757.log.gz 2024-09-22T01:02:06.834 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112981.log.gz 2024-09-22T01:02:06.834 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.95991.log.gz 2024-09-22T01:02:06.834 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77102.log.gz 2024-09-22T01:02:06.834 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.76796.log.gz 2024-09-22T01:02:06.834 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.93730.log.gz 2024-09-22T01:02:06.834 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116793.log.gz 2024-09-22T01:02:06.834 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80106.log.gz 2024-09-22T01:02:06.834 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.73843.log.gz 2024-09-22T01:02:06.834 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.62052.log.gz 2024-09-22T01:02:06.834 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56665.log.gz 2024-09-22T01:02:06.834 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32877.log.gz 2024-09-22T01:02:06.835 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98934.log.gz 2024-09-22T01:02:06.835 INFO:teuthology.orchestra.run.smithi060.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.86496.log.gz 2024-09-22T01:02:06.835 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140482.log.gz 2024-09-22T01:02:06.835 INFO:teuthology.orchestra.run.smithi060.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.74900.log.gz 2024-09-22T01:02:06.835 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56371.log.gz 90.1% 2024-09-22T01:02:06.835 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.133907.log.gz 2024-09-22T01:02:06.835 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37520.log.gz 2024-09-22T01:02:06.835 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109188.log.gz 2024-09-22T01:02:06.835 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37141.log.gz 2024-09-22T01:02:06.835 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.96042.log.gz 2024-09-22T01:02:06.835 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.132921.log.gz 2024-09-22T01:02:06.835 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36639.log.gz 2024-09-22T01:02:06.835 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127436.log.gz 2024-09-22T01:02:06.835 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90487.log.gz 2024-09-22T01:02:06.835 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68933.log.gz 2024-09-22T01:02:06.835 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.64541.log.gz 2024-09-22T01:02:06.835 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49051.log.gz 2024-09-22T01:02:06.836 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78614.log.gz 2024-09-22T01:02:06.836 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102852.log.gz 2024-09-22T01:02:06.836 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.103272.log.gz 2024-09-22T01:02:06.836 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133219.log.gz 2024-09-22T01:02:06.836 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136178.log.gz 2024-09-22T01:02:06.836 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45852.log.gz 2024-09-22T01:02:06.836 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60278.log.gz 2024-09-22T01:02:06.836 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137230.log.gz 2024-09-22T01:02:06.836 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113568.log.gz 2024-09-22T01:02:06.836 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.98492.log.gz 2024-09-22T01:02:06.836 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116876.log.gz 2024-09-22T01:02:06.836 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67574.log.gz 2024-09-22T01:02:06.836 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35839.log.gz 2024-09-22T01:02:06.836 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68347.log.gz 2024-09-22T01:02:06.836 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42521.log.gz 2024-09-22T01:02:06.837 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51064.log.gz 2024-09-22T01:02:06.837 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138406.log.gz 2024-09-22T01:02:06.837 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34958.log.gz 2024-09-22T01:02:06.837 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.104740.log.gz 2024-09-22T01:02:06.837 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59521.log.gz 2024-09-22T01:02:06.837 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45851.log.gz 2024-09-22T01:02:06.837 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102011.log.gz 2024-09-22T01:02:06.837 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.89430.log.gz 2024-09-22T01:02:06.837 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118329.log.gz 90.0% 2024-09-22T01:02:06.837 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.103123.log.gz 2024-09-22T01:02:06.837 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49094.log.gz 2024-09-22T01:02:06.837 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103174.log.gz 2024-09-22T01:02:06.837 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57253.log.gz 2024-09-22T01:02:06.837 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42331.log.gz 2024-09-22T01:02:06.838 INFO:teuthology.orchestra.run.smithi060.stderr: 58.3% -- replaced with /var/log/ceph/cluster2-client.mirror.30228.log.gz 2024-09-22T01:02:06.838 INFO:teuthology.orchestra.run.smithi060.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.119901.log.gz 2024-09-22T01:02:06.838 INFO:teuthology.orchestra.run.smithi060.stderr: 56.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30057.log.gz 2024-09-22T01:02:06.838 INFO:teuthology.orchestra.run.smithi060.stderr: 91.1% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38089.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.108723.log.gz 2024-09-22T01:02:06.838 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.838 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.88110.log.gz 2024-09-22T01:02:06.838 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71033.log.gz 2024-09-22T01:02:06.838 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.105265.log.gz 2024-09-22T01:02:06.838 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.130320.log.gz 2024-09-22T01:02:06.838 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126031.log.gz 2024-09-22T01:02:06.838 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109255.log.gz 2024-09-22T01:02:06.838 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113881.log.gz 2024-09-22T01:02:06.838 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114030.log.gz 2024-09-22T01:02:06.838 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.129413.log.gz 2024-09-22T01:02:06.839 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.106776.log.gz 2024-09-22T01:02:06.839 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.105374.log.gz 2024-09-22T01:02:06.839 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77793.log.gz 2024-09-22T01:02:06.839 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.95127.log.gz 2024-09-22T01:02:06.839 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.39728.log.gz 2024-09-22T01:02:06.839 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89010.log.gz 2024-09-22T01:02:06.839 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53878.log.gz 2024-09-22T01:02:06.839 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137482.log.gz 2024-09-22T01:02:06.839 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.132155.log.gz 2024-09-22T01:02:06.839 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.131880.log.gz 2024-09-22T01:02:06.839 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.37479.log.gz 2024-09-22T01:02:06.839 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136031.log.gz 2024-09-22T01:02:06.839 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90623.log.gz 2024-09-22T01:02:06.839 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124601.log.gz 2024-09-22T01:02:06.839 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.96151.log.gz 2024-09-22T01:02:06.840 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65518.log.gz 2024-09-22T01:02:06.840 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90477.log.gz 2024-09-22T01:02:06.840 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.89009.log.gz 2024-09-22T01:02:06.840 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136429.log.gz 2024-09-22T01:02:06.840 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53598.log.gz 2024-09-22T01:02:06.840 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102200.log.gz 2024-09-22T01:02:06.840 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.99635.log.gz 2024-09-22T01:02:06.840 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68137.log.gz 2024-09-22T01:02:06.840 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.104090.log.gz 2024-09-22T01:02:06.840 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79557.log.gz 2024-09-22T01:02:06.840 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.120481.log.gz 2024-09-22T01:02:06.840 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.110037.log.gz 2024-09-22T01:02:06.840 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.132492.log.gz 2024-09-22T01:02:06.840 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133138.log.gz 2024-09-22T01:02:06.840 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.135842.log.gz 2024-09-22T01:02:06.841 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122437.log.gz 2024-09-22T01:02:06.841 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.95943.log.gz 2024-09-22T01:02:06.841 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57968.log.gz 2024-09-22T01:02:06.841 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.121193.log.gz 2024-09-22T01:02:06.841 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81050.log.gz 2024-09-22T01:02:06.841 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35399.log.gz 2024-09-22T01:02:06.841 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.82689.log.gz 2024-09-22T01:02:06.841 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102912.log.gz 89.9% 2024-09-22T01:02:06.841 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.86813.log.gz 2024-09-22T01:02:06.841 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117833.log.gz 2024-09-22T01:02:06.841 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.97604.log.gz 2024-09-22T01:02:06.841 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70783.log.gz 2024-09-22T01:02:06.841 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64297.log.gz 2024-09-22T01:02:06.841 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54554.log.gz 2024-09-22T01:02:06.842 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67152.log.gz 2024-09-22T01:02:06.842 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115811.log.gz 2024-09-22T01:02:06.842 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.98823.log.gz 2024-09-22T01:02:06.842 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42460.log.gz 2024-09-22T01:02:06.842 INFO:teuthology.orchestra.run.smithi060.stderr: 67.5% -- replaced with /var/log/ceph/cluster1-client.mirror.119559.log.gz 2024-09-22T01:02:06.842 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46273.log.gz 2024-09-22T01:02:06.842 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.135966.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.87132.log.gz 2024-09-22T01:02:06.842 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.842 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.66623.log.gz 2024-09-22T01:02:06.842 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.86040.log.gz 2024-09-22T01:02:06.842 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.61900.log.gz 2024-09-22T01:02:06.842 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80818.log.gz 2024-09-22T01:02:06.842 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.121599.log.gz 90.7% 2024-09-22T01:02:06.842 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.53015.log.gz 2024-09-22T01:02:06.842 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35441.log.gz 2024-09-22T01:02:06.842 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92344.log.gz 2024-09-22T01:02:06.843 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36849.log.gz 2024-09-22T01:02:06.843 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54956.log.gz 2024-09-22T01:02:06.843 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.admin.140487.log.gz 2024-09-22T01:02:06.843 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34411.log.gz 2024-09-22T01:02:06.843 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125379.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.68808.log.gz 2024-09-22T01:02:06.843 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.843 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110290.log.gz 2024-09-22T01:02:06.843 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.81825.log.gz 2024-09-22T01:02:06.843 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45771.log.gz 2024-09-22T01:02:06.843 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58221.log.gz 2024-09-22T01:02:06.843 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.121706.log.gz 2024-09-22T01:02:06.843 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67213.log.gz 2024-09-22T01:02:06.843 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.37836.log.gz 2024-09-22T01:02:06.843 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.95128.log.gz 2024-09-22T01:02:06.843 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.101487.log.gz 2024-09-22T01:02:06.843 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126006.log.gz 2024-09-22T01:02:06.844 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103419.log.gz 2024-09-22T01:02:06.844 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.76819.log.gz 2024-09-22T01:02:06.844 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79851.log.gz 2024-09-22T01:02:06.844 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34872.log.gz 2024-09-22T01:02:06.844 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104531.log.gz 2024-09-22T01:02:06.844 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77876.log.gz 2024-09-22T01:02:06.844 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123932.log.gz 2024-09-22T01:02:06.844 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.138157.log.gz 2024-09-22T01:02:06.844 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.106119.log.gz 2024-09-22T01:02:06.844 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.97490.log.gz 2024-09-22T01:02:06.844 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86066.log.gz 2024-09-22T01:02:06.844 INFO:teuthology.orchestra.run.smithi060.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64843.log.gz 2024-09-22T01:02:06.844 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.101277.log.gz 2024-09-22T01:02:06.844 INFO:teuthology.orchestra.run.smithi060.stderr: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.30194.log.gz 2024-09-22T01:02:06.844 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76753.log.gz 2024-09-22T01:02:06.845 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59651.log.gz 2024-09-22T01:02:06.845 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56035.log.gz 2024-09-22T01:02:06.845 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.101318.log.gz 2024-09-22T01:02:06.845 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46697.log.gz 2024-09-22T01:02:06.845 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135883.log.gz 2024-09-22T01:02:06.845 INFO:teuthology.orchestra.run.smithi060.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-client.admin.75175.log.gz 2024-09-22T01:02:06.845 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80482.log.gz 2024-09-22T01:02:06.845 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35921.log.gz 2024-09-22T01:02:06.845 INFO:teuthology.orchestra.run.smithi060.stderr: 80.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53688.log.gz 87.4% 2024-09-22T01:02:06.845 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.117878.log.gz 2024-09-22T01:02:06.845 INFO:teuthology.orchestra.run.smithi060.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.94108.log.gz 90.0% 2024-09-22T01:02:06.845 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.82076.log.gz 2024-09-22T01:02:06.845 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.85899.log.gz 2024-09-22T01:02:06.845 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126451.log.gz 2024-09-22T01:02:06.845 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126743.log.gz 2024-09-22T01:02:06.845 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.96284.log.gz 2024-09-22T01:02:06.846 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68204.log.gz 2024-09-22T01:02:06.846 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44045.log.gz 2024-09-22T01:02:06.846 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.53387.log.gz 2024-09-22T01:02:06.846 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44761.log.gz 2024-09-22T01:02:06.846 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35562.log.gz 2024-09-22T01:02:06.846 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57841.log.gz 2024-09-22T01:02:06.846 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121514.log.gz 2024-09-22T01:02:06.846 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.48336.log.gz 2024-09-22T01:02:06.846 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.96642.log.gz 2024-09-22T01:02:06.846 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.51400.log.gz 2024-09-22T01:02:06.846 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42289.log.gz 2024-09-22T01:02:06.846 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70656.log.gz 2024-09-22T01:02:06.846 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113820.log.gz 2024-09-22T01:02:06.846 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.97034.log.gz 2024-09-22T01:02:06.846 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86877.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.89739.log.gz 2024-09-22T01:02:06.847 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.847 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65195.log.gz 2024-09-22T01:02:06.847 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100689.log.gz 2024-09-22T01:02:06.847 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70302.log.gz 2024-09-22T01:02:06.847 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125169.log.gz 2024-09-22T01:02:06.847 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.73564.log.gz 2024-09-22T01:02:06.847 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.86582.log.gz 2024-09-22T01:02:06.847 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93770.log.gz 90.1% 2024-09-22T01:02:06.847 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.127584.log.gz 2024-09-22T01:02:06.847 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33418.log.gz 2024-09-22T01:02:06.847 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.51672.log.gz 2024-09-22T01:02:06.847 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57213.log.gz 2024-09-22T01:02:06.847 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.31687.log.gz 2024-09-22T01:02:06.847 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.85043.log.gz 2024-09-22T01:02:06.847 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36341.log.gz 90.4% 2024-09-22T01:02:06.847 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.46444.log.gz 2024-09-22T01:02:06.847 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58556.log.gz 2024-09-22T01:02:06.848 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68201.log.gz 2024-09-22T01:02:06.848 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109168.log.gz 2024-09-22T01:02:06.848 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116443.log.gz 2024-09-22T01:02:06.848 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.61899.log.gz 2024-09-22T01:02:06.848 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112536.log.gz 2024-09-22T01:02:06.848 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80209.log.gz 2024-09-22T01:02:06.848 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32009.log.gz 2024-09-22T01:02:06.848 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91043.log.gz 2024-09-22T01:02:06.848 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.131982.log.gz 2024-09-22T01:02:06.848 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65498.log.gz 2024-09-22T01:02:06.848 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125401.log.gz 2024-09-22T01:02:06.848 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37268.log.gz 2024-09-22T01:02:06.848 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113924.log.gz 2024-09-22T01:02:06.848 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35772.log.gz 2024-09-22T01:02:06.848 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46398.log.gz 2024-09-22T01:02:06.848 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.31925.log.gz 2024-09-22T01:02:06.848 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.108810.log.gz 2024-09-22T01:02:06.849 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71032.log.gz 2024-09-22T01:02:06.849 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73357.log.gz 2024-09-22T01:02:06.849 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.106120.log.gz 2024-09-22T01:02:06.849 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105923.log.gz 2024-09-22T01:02:06.849 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.77917.log.gz 90.4% 2024-09-22T01:02:06.849 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.137187.log.gz 2024-09-22T01:02:06.849 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.38914.log.gz 2024-09-22T01:02:06.849 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47201.log.gz 2024-09-22T01:02:06.849 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80211.log.gz 2024-09-22T01:02:06.849 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.48797.log.gz 2024-09-22T01:02:06.849 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113081.log.gz 2024-09-22T01:02:06.849 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62319.log.gz 2024-09-22T01:02:06.849 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49753.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.32917.log.gz 2024-09-22T01:02:06.849 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.849 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124560.log.gz 2024-09-22T01:02:06.849 INFO:teuthology.orchestra.run.smithi060.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.97008.log.gz 2024-09-22T01:02:06.850 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34618.log.gz 2024-09-22T01:02:06.850 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62697.log.gz 2024-09-22T01:02:06.850 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80105.log.gz 2024-09-22T01:02:06.850 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80775.log.gz 2024-09-22T01:02:06.850 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.132304.log.gz 2024-09-22T01:02:06.850 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.31096.log.gz 2024-09-22T01:02:06.850 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124642.log.gz 2024-09-22T01:02:06.850 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46951.log.gz 2024-09-22T01:02:06.850 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65239.log.gz 2024-09-22T01:02:06.850 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.39617.log.gz 2024-09-22T01:02:06.850 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100478.log.gz 2024-09-22T01:02:06.850 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116589.log.gz 2024-09-22T01:02:06.850 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56833.log.gz 2024-09-22T01:02:06.850 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128228.log.gz 2024-09-22T01:02:06.850 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40116.log.gz 2024-09-22T01:02:06.850 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.76184.log.gz 2024-09-22T01:02:06.851 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103393.log.gz 2024-09-22T01:02:06.851 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59660.log.gz 2024-09-22T01:02:06.851 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113357.log.gz 2024-09-22T01:02:06.851 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.76901.log.gz 2024-09-22T01:02:06.851 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.31814.log.gz 2024-09-22T01:02:06.851 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113671.log.gz 2024-09-22T01:02:06.851 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104046.log.gz 2024-09-22T01:02:06.851 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56563.log.gz 2024-09-22T01:02:06.851 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43454.log.gz 2024-09-22T01:02:06.851 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98239.log.gz 2024-09-22T01:02:06.851 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79009.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.43364.log.gz 2024-09-22T01:02:06.851 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.851 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.97464.log.gz 2024-09-22T01:02:06.851 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.54510.log.gz 2024-09-22T01:02:06.851 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41954.log.gz 2024-09-22T01:02:06.851 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.35798.log.gz 2024-09-22T01:02:06.852 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.107242.log.gz 2024-09-22T01:02:06.852 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92093.log.gz 2024-09-22T01:02:06.852 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68514.log.gz 2024-09-22T01:02:06.852 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.107239.log.gz 2024-09-22T01:02:06.852 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.105309.log.gz 2024-09-22T01:02:06.852 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71562.log.gz 2024-09-22T01:02:06.852 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127647.log.gz 2024-09-22T01:02:06.852 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57001.log.gz 2024-09-22T01:02:06.852 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35584.log.gz 2024-09-22T01:02:06.852 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.106775.log.gz 2024-09-22T01:02:06.852 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.36469.log.gz 2024-09-22T01:02:06.852 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56122.log.gz 2024-09-22T01:02:06.852 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81051.log.gz 2024-09-22T01:02:06.852 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.89529.log.gz 2024-09-22T01:02:06.853 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87509.log.gz 2024-09-22T01:02:06.853 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.108468.log.gz 2024-09-22T01:02:06.853 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33882.log.gz 2024-09-22T01:02:06.853 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48251.log.gz 2024-09-22T01:02:06.853 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36890.log.gz 2024-09-22T01:02:06.853 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54024.log.gz 2024-09-22T01:02:06.853 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138072.log.gz 2024-09-22T01:02:06.853 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.86978.log.gz 90.0% 2024-09-22T01:02:06.853 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.125820.log.gz 2024-09-22T01:02:06.853 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36971.log.gz 2024-09-22T01:02:06.853 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.86814.log.gz 2024-09-22T01:02:06.853 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112832.log.gz 2024-09-22T01:02:06.853 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.102997.log.gz 2024-09-22T01:02:06.853 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58051.log.gz 2024-09-22T01:02:06.853 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82287.log.gz 2024-09-22T01:02:06.853 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.38695.log.gz 2024-09-22T01:02:06.853 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34578.log.gz 2024-09-22T01:02:06.854 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.82186.log.gz 2024-09-22T01:02:06.854 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.91084.log.gz 2024-09-22T01:02:06.854 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.104676.log.gz 2024-09-22T01:02:06.854 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58598.log.gz 2024-09-22T01:02:06.854 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.95572.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.37663.log.gz 2024-09-22T01:02:06.854 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.854 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.34997.log.gz 2024-09-22T01:02:06.854 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.89109.log.gz 2024-09-22T01:02:06.854 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104847.log.gz 2024-09-22T01:02:06.854 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.108179.log.gz 2024-09-22T01:02:06.854 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.31514.log.gz 2024-09-22T01:02:06.854 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.80272.log.gz 2024-09-22T01:02:06.854 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44593.log.gz 87.8% 2024-09-22T01:02:06.854 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.73751.log.gz 2024-09-22T01:02:06.854 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120734.log.gz 2024-09-22T01:02:06.854 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.48176.log.gz 2024-09-22T01:02:06.855 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.37181.log.gz 2024-09-22T01:02:06.855 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113711.log.gz 2024-09-22T01:02:06.855 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35503.log.gz 2024-09-22T01:02:06.855 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.65307.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.70512.log.gz 2024-09-22T01:02:06.855 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.855 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102431.log.gz 2024-09-22T01:02:06.855 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114728.log.gz 2024-09-22T01:02:06.855 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.30767.log.gz 2024-09-22T01:02:06.855 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77918.log.gz 2024-09-22T01:02:06.855 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81548.log.gz 2024-09-22T01:02:06.855 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.51323.log.gz 2024-09-22T01:02:06.855 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98785.log.gz 2024-09-22T01:02:06.855 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.135800.log.gz 2024-09-22T01:02:06.855 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.133220.log.gz 2024-09-22T01:02:06.855 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46083.log.gz 2024-09-22T01:02:06.855 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.51907.log.gz 2024-09-22T01:02:06.855 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.71707.log.gz 2024-09-22T01:02:06.856 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45223.log.gz 2024-09-22T01:02:06.856 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122519.log.gz 2024-09-22T01:02:06.856 INFO:teuthology.orchestra.run.smithi060.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52968.log.gz 2024-09-22T01:02:06.856 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55107.log.gz 2024-09-22T01:02:06.856 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.107257.log.gz 2024-09-22T01:02:06.856 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70616.log.gz 2024-09-22T01:02:06.856 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137610.log.gz 2024-09-22T01:02:06.856 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.125693.log.gz 2024-09-22T01:02:06.856 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.98322.log.gz 2024-09-22T01:02:06.856 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55275.log.gz 2024-09-22T01:02:06.856 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57254.log.gz 2024-09-22T01:02:06.856 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.106263.log.gz 2024-09-22T01:02:06.856 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124645.log.gz 2024-09-22T01:02:06.856 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.140532.log.gz 2024-09-22T01:02:06.857 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.120966.log.gz 2024-09-22T01:02:06.857 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123513.log.gz 2024-09-22T01:02:06.857 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.71666.log.gz 2024-09-22T01:02:06.857 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55994.log.gz 2024-09-22T01:02:06.857 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32668.log.gz 2024-09-22T01:02:06.857 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.105478.log.gz 2024-09-22T01:02:06.857 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118954.log.gz 2024-09-22T01:02:06.857 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.131855.log.gz 2024-09-22T01:02:06.857 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55531.log.gz 2024-09-22T01:02:06.857 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.79430.log.gz 2024-09-22T01:02:06.857 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.75262.log.gz 2024-09-22T01:02:06.857 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.121535.log.gz 2024-09-22T01:02:06.857 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122544.log.gz 2024-09-22T01:02:06.857 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.117451.log.gz 2024-09-22T01:02:06.857 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41457.log.gz 2024-09-22T01:02:06.857 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60026.log.gz 2024-09-22T01:02:06.857 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77241.log.gz 2024-09-22T01:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133823.log.gz 2024-09-22T01:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115921.log.gz 2024-09-22T01:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58555.log.gz 2024-09-22T01:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70364.log.gz 2024-09-22T01:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80735.log.gz 2024-09-22T01:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52357.log.gz 2024-09-22T01:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89008.log.gz 2024-09-22T01:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59439.log.gz 2024-09-22T01:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78046.log.gz 2024-09-22T01:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46317.log.gz 2024-09-22T01:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.96041.log.gz 2024-09-22T01:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65639.log.gz 2024-09-22T01:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.54214.log.gz 90.2% 2024-09-22T01:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.71563.log.gz 2024-09-22T01:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.62050.log.gz 2024-09-22T01:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113672.log.gz 2024-09-22T01:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.93813.log.gz 2024-09-22T01:02:06.859 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.82750.log.gz 2024-09-22T01:02:06.859 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.48293.log.gz 2024-09-22T01:02:06.859 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82161.log.gz 2024-09-22T01:02:06.859 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41571.log.gz 2024-09-22T01:02:06.859 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.32750.log.gz 2024-09-22T01:02:06.859 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.31201.log.gz 2024-09-22T01:02:06.859 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.97323.log.gz 2024-09-22T01:02:06.859 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132920.log.gz 2024-09-22T01:02:06.859 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124432.log.gz 2024-09-22T01:02:06.859 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.91110.log.gz 2024-09-22T01:02:06.859 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100874.log.gz 2024-09-22T01:02:06.859 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103752.log.gz 2024-09-22T01:02:06.859 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.91948.log.gz 2024-09-22T01:02:06.859 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.77959.log.gz 2024-09-22T01:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129399.log.gz 2024-09-22T01:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.109741.log.gz 2024-09-22T01:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133824.log.gz 2024-09-22T01:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.76224.log.gz 2024-09-22T01:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91924.log.gz 2024-09-22T01:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.114887.log.gz 2024-09-22T01:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.30143.log.gz 2024-09-22T01:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102432.log.gz 2024-09-22T01:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr: 67.5% -- replaced with /var/log/ceph/cluster1-client.mirror.120243.log.gz 2024-09-22T01:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.131944.log.gz 2024-09-22T01:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57506.log.gz 2024-09-22T01:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46826.log.gz 2024-09-22T01:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121878.log.gz 2024-09-22T01:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112770.log.gz 2024-09-22T01:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58327.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.98640.log.gz 2024-09-22T01:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.137315.log.gz 2024-09-22T01:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.113501.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.31647.log.gz 2024-09-22T01:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35377.log.gz 2024-09-22T01:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101442.log.gz 2024-09-22T01:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.81909.log.gz 2024-09-22T01:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44679.log.gz 2024-09-22T01:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121601.log.gz 2024-09-22T01:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35271.log.gz 2024-09-22T01:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113399.log.gz 2024-09-22T01:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71538.log.gz 2024-09-22T01:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114509.log.gz 2024-09-22T01:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125754.log.gz 2024-09-22T01:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53814.log.gz 2024-09-22T01:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.87806.log.gz 2024-09-22T01:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.52874.log.gz 2024-09-22T01:02:06.862 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.102535.log.gz 2024-09-22T01:02:06.862 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.87277.log.gz 2024-09-22T01:02:06.862 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45391.log.gz 2024-09-22T01:02:06.862 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102828.log.gz 2024-09-22T01:02:06.862 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.104427.log.gz 2024-09-22T01:02:06.862 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.81576.log.gz 2024-09-22T01:02:06.862 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101907.log.gz 2024-09-22T01:02:06.862 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.99694.log.gz 2024-09-22T01:02:06.862 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.90960.log.gz 2024-09-22T01:02:06.862 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.62752.log.gz 2024-09-22T01:02:06.862 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43560.log.gz 2024-09-22T01:02:06.862 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58432.log.gz 2024-09-22T01:02:06.862 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.113610.log.gz 2024-09-22T01:02:06.862 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116399.log.gz 2024-09-22T01:02:06.862 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105503.log.gz 2024-09-22T01:02:06.862 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102124.log.gz 2024-09-22T01:02:06.862 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.120413.log.gz 2024-09-22T01:02:06.863 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.91419.log.gz 2024-09-22T01:02:06.863 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134033.log.gz 2024-09-22T01:02:06.863 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65159.log.gz 2024-09-22T01:02:06.863 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.61467.log.gz 2024-09-22T01:02:06.863 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.36342.log.gz 2024-09-22T01:02:06.863 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.110312.log.gz 2024-09-22T01:02:06.863 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.87025.log.gz 2024-09-22T01:02:06.863 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.138281.log.gz 2024-09-22T01:02:06.863 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90689.log.gz 2024-09-22T01:02:06.863 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.52646.log.gz 2024-09-22T01:02:06.863 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.139937.log.gz 2024-09-22T01:02:06.863 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45687.log.gz 2024-09-22T01:02:06.863 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53155.log.gz 2024-09-22T01:02:06.863 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.107114.log.gz 2024-09-22T01:02:06.863 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.48167.log.gz 90.1% 2024-09-22T01:02:06.864 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.100604.log.gz 2024-09-22T01:02:06.864 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.48211.log.gz 2024-09-22T01:02:06.864 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.81366.log.gz 87.6% 2024-09-22T01:02:06.864 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.73766.log.gz 2024-09-22T01:02:06.864 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.31055.log.gz 2024-09-22T01:02:06.864 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.47117.log.gz 2024-09-22T01:02:06.864 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.33697.log.gz 2024-09-22T01:02:06.864 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63540.log.gz 2024-09-22T01:02:06.864 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63613.log.gz 2024-09-22T01:02:06.864 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.87467.log.gz 2024-09-22T01:02:06.864 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.81489.log.gz 2024-09-22T01:02:06.864 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70152.log.gz 2024-09-22T01:02:06.864 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54340.log.gz 2024-09-22T01:02:06.864 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113502.log.gz 2024-09-22T01:02:06.864 INFO:teuthology.orchestra.run.smithi060.stderr: 66.8% 91.2% -- replaced with /var/log/ceph/cluster1-client.mirror.97417.log.gz 2024-09-22T01:02:06.865 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.108381.log.gz 2024-09-22T01:02:06.865 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100476.log.gz 2024-09-22T01:02:06.865 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120607.log.gz 2024-09-22T01:02:06.865 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127015.log.gz 2024-09-22T01:02:06.865 INFO:teuthology.orchestra.run.smithi060.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.108904.log.gz 2024-09-22T01:02:06.865 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114090.log.gz 2024-09-22T01:02:06.865 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 87.2% 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.68834.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.96491.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.53216.log.gz 2024-09-22T01:02:06.865 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.865 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.865 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43538.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.120736.log.gz 2024-09-22T01:02:06.865 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.865 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67047.log.gz 2024-09-22T01:02:06.865 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.65263.log.gz 2024-09-22T01:02:06.865 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.48127.log.gz 2024-09-22T01:02:06.865 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36974.log.gz 2024-09-22T01:02:06.865 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80315.log.gz 2024-09-22T01:02:06.866 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80607.log.gz 2024-09-22T01:02:06.866 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82690.log.gz 2024-09-22T01:02:06.866 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76353.log.gz 2024-09-22T01:02:06.866 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80357.log.gz 2024-09-22T01:02:06.866 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.33907.log.gz 2024-09-22T01:02:06.866 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69042.log.gz 2024-09-22T01:02:06.866 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.103418.log.gz 2024-09-22T01:02:06.866 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91840.log.gz 2024-09-22T01:02:06.866 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.79724.log.gz 2024-09-22T01:02:06.866 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.96168.log.gz 2024-09-22T01:02:06.866 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42010.log.gz 2024-09-22T01:02:06.866 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.61987.log.gz 2024-09-22T01:02:06.866 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.65702.log.gz 2024-09-22T01:02:06.866 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78843.log.gz 2024-09-22T01:02:06.867 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.88331.log.gz 2024-09-22T01:02:06.867 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47916.log.gz 2024-09-22T01:02:06.867 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65603.log.gz 2024-09-22T01:02:06.867 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.95286.log.gz 87.7% 2024-09-22T01:02:06.867 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.73399.log.gz 2024-09-22T01:02:06.867 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36929.log.gz 2024-09-22T01:02:06.867 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82288.log.gz 2024-09-22T01:02:06.867 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.64992.log.gz 2024-09-22T01:02:06.867 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.32413.log.gz 2024-09-22T01:02:06.867 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.95438.log.gz 2024-09-22T01:02:06.867 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.50441.log.gz 2024-09-22T01:02:06.867 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.128913.log.gz 2024-09-22T01:02:06.868 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112979.log.gz 2024-09-22T01:02:06.868 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.120434.log.gz 2024-09-22T01:02:06.868 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70868.log.gz 2024-09-22T01:02:06.868 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.93729.log.gz 2024-09-22T01:02:06.868 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.128600.log.gz 2024-09-22T01:02:06.868 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136033.log.gz 2024-09-22T01:02:06.868 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.38855.log.gz 2024-09-22T01:02:06.868 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31496.log.gz 2024-09-22T01:02:06.868 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.50820.log.gz 2024-09-22T01:02:06.868 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.61468.log.gz 2024-09-22T01:02:06.868 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86686.log.gz 2024-09-22T01:02:06.868 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37856.log.gz 2024-09-22T01:02:06.868 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65665.log.gz 2024-09-22T01:02:06.868 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73394.log.gz 2024-09-22T01:02:06.868 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65471.log.gz 2024-09-22T01:02:06.868 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58012.log.gz 2024-09-22T01:02:06.869 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77962.log.gz 2024-09-22T01:02:06.869 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70934.log.gz 2024-09-22T01:02:06.869 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85952.log.gz 2024-09-22T01:02:06.869 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.30545.log.gz 2024-09-22T01:02:06.869 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121115.log.gz 2024-09-22T01:02:06.869 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.31176.log.gz 2024-09-22T01:02:06.869 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91307.log.gz 2024-09-22T01:02:06.869 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.31537.log.gz 2024-09-22T01:02:06.869 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.114660.log.gz 2024-09-22T01:02:06.869 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.100730.log.gz 2024-09-22T01:02:06.869 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90622.log.gz 2024-09-22T01:02:06.869 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103587.log.gz 2024-09-22T01:02:06.869 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44720.log.gz 2024-09-22T01:02:06.869 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.97209.log.gz 2024-09-22T01:02:06.869 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53329.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.97095.log.gz 2024-09-22T01:02:06.870 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.870 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55702.log.gz 2024-09-22T01:02:06.870 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102433.log.gz 2024-09-22T01:02:06.870 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.103878.log.gz 2024-09-22T01:02:06.870 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132534.log.gz 2024-09-22T01:02:06.870 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78297.log.gz 2024-09-22T01:02:06.870 INFO:teuthology.orchestra.run.smithi060.stderr: 67.3% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63768.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.104530.log.gz 2024-09-22T01:02:06.870 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.870 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32112.log.gz 2024-09-22T01:02:06.870 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.41746.log.gz 2024-09-22T01:02:06.870 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.76566.log.gz 2024-09-22T01:02:06.870 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67573.log.gz 2024-09-22T01:02:06.870 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.46146.log.gz 2024-09-22T01:02:06.870 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40176.log.gz 2024-09-22T01:02:06.871 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57295.log.gz 90.1% 2024-09-22T01:02:06.871 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.126596.log.gz 2024-09-22T01:02:06.871 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88792.log.gz 2024-09-22T01:02:06.871 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58933.log.gz 2024-09-22T01:02:06.871 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.80210.log.gz 2024-09-22T01:02:06.871 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.105162.log.gz 2024-09-22T01:02:06.871 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.88891.log.gz 2024-09-22T01:02:06.871 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.68202.log.gz 2024-09-22T01:02:06.871 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32372.log.gz 2024-09-22T01:02:06.871 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103209.log.gz 2024-09-22T01:02:06.871 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65433.log.gz 2024-09-22T01:02:06.872 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46992.log.gz 2024-09-22T01:02:06.872 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.87434.log.gz 2024-09-22T01:02:06.872 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78547.log.gz 2024-09-22T01:02:06.872 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42496.log.gz 2024-09-22T01:02:06.872 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.61566.log.gz 2024-09-22T01:02:06.872 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.118386.log.gz 2024-09-22T01:02:06.872 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70866.log.gz 2024-09-22T01:02:06.872 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.30524.log.gz 2024-09-22T01:02:06.872 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100962.log.gz 2024-09-22T01:02:06.872 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36889.log.gz 2024-09-22T01:02:06.872 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.95890.log.gz 2024-09-22T01:02:06.872 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.137399.log.gz 2024-09-22T01:02:06.872 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113779.log.gz 2024-09-22T01:02:06.873 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60613.log.gz 2024-09-22T01:02:06.873 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35711.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.102850.log.gz 2024-09-22T01:02:06.873 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.873 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105057.log.gz 2024-09-22T01:02:06.873 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121365.log.gz 2024-09-22T01:02:06.873 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.133748.log.gz 2024-09-22T01:02:06.873 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58808.log.gz 2024-09-22T01:02:06.873 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130373.log.gz 2024-09-22T01:02:06.873 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.48841.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.71478.log.gz 2024-09-22T01:02:06.873 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.873 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56565.log.gz 2024-09-22T01:02:06.873 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.96981.log.gz 2024-09-22T01:02:06.873 INFO:teuthology.orchestra.run.smithi060.stderr: 68.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52854.log.gz 2024-09-22T01:02:06.874 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70092.log.gz 2024-09-22T01:02:06.874 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68976.log.gz 2024-09-22T01:02:06.874 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.61497.log.gz 2024-09-22T01:02:06.874 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.73753.log.gz 2024-09-22T01:02:06.874 INFO:teuthology.orchestra.run.smithi060.stderr: 80.5% -- replaced with /var/log/ceph/cluster1-client.mirror.132093.log.gz 2024-09-22T01:02:06.874 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123720.log.gz 2024-09-22T01:02:06.874 INFO:teuthology.orchestra.run.smithi060.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.119673.log.gz 2024-09-22T01:02:06.874 INFO:teuthology.orchestra.run.smithi060.stderr: 80.5% -- replaced with /var/log/ceph/cluster1-client.mirror.120480.log.gz 2024-09-22T01:02:06.874 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.50205.log.gz 2024-09-22T01:02:06.874 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35398.log.gz 2024-09-22T01:02:06.874 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54470.log.gz 2024-09-22T01:02:06.874 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60194.log.gz 2024-09-22T01:02:06.874 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.120772.log.gz 2024-09-22T01:02:06.874 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.39614.log.gz 2024-09-22T01:02:06.875 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.87930.log.gz 2024-09-22T01:02:06.875 INFO:teuthology.orchestra.run.smithi060.stderr: 80.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53435.log.gz 2024-09-22T01:02:06.875 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.50818.log.gz 2024-09-22T01:02:06.875 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.117940.log.gz 2024-09-22T01:02:06.875 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127836.log.gz 2024-09-22T01:02:06.875 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37993.log.gz 2024-09-22T01:02:06.875 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32709.log.gz 2024-09-22T01:02:06.875 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70404.log.gz 2024-09-22T01:02:06.875 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.37603.log.gz 2024-09-22T01:02:06.875 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.106586.log.gz 2024-09-22T01:02:06.875 INFO:teuthology.orchestra.run.smithi060.stderr: 19.6% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.30297.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.39616.log.gz 2024-09-22T01:02:06.875 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.876 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.98168.log.gz 2024-09-22T01:02:06.876 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.38919.log.gz 2024-09-22T01:02:06.876 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125484.log.gz 2024-09-22T01:02:06.876 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116297.log.gz 2024-09-22T01:02:06.876 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.62743.log.gz 88.1% 2024-09-22T01:02:06.876 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.63028.log.gz 2024-09-22T01:02:06.876 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55867.log.gz 2024-09-22T01:02:06.876 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.74719.log.gz 2024-09-22T01:02:06.876 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.50819.log.gz 2024-09-22T01:02:06.876 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103670.log.gz 2024-09-22T01:02:06.876 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.64130.log.gz 2024-09-22T01:02:06.876 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.62741.log.gz 2024-09-22T01:02:06.876 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.87742.log.gz 2024-09-22T01:02:06.876 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.48798.log.gz 2024-09-22T01:02:06.876 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.117877.log.gz 2024-09-22T01:02:06.877 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113712.log.gz 2024-09-22T01:02:06.877 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126491.log.gz 2024-09-22T01:02:06.877 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.98743.log.gz 2024-09-22T01:02:06.877 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.51785.log.gz 2024-09-22T01:02:06.877 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58093.log.gz 2024-09-22T01:02:06.877 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66582.log.gz 2024-09-22T01:02:06.877 INFO:teuthology.orchestra.run.smithi060.stderr: 67.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53082.log.gz 91.8% 2024-09-22T01:02:06.877 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.84612.log.gz 2024-09-22T01:02:06.877 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46825.log.gz 2024-09-22T01:02:06.877 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86522.log.gz 2024-09-22T01:02:06.877 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.63587.log.gz 2024-09-22T01:02:06.877 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.139775.log.gz 2024-09-22T01:02:06.877 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81660.log.gz 2024-09-22T01:02:06.877 INFO:teuthology.orchestra.run.smithi060.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120015.log.gz 2024-09-22T01:02:06.877 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56288.log.gz 2024-09-22T01:02:06.877 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.48504.log.gz 2024-09-22T01:02:06.877 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.138073.log.gz 2024-09-22T01:02:06.878 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79053.log.gz 2024-09-22T01:02:06.878 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.118016.log.gz 2024-09-22T01:02:06.878 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110117.log.gz 2024-09-22T01:02:06.878 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.132007.log.gz 2024-09-22T01:02:06.878 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79723.log.gz 2024-09-22T01:02:06.878 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.104951.log.gz 2024-09-22T01:02:06.878 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.132877.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.84419.log.gz 2024-09-22T01:02:06.878 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.878 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99575.log.gz 2024-09-22T01:02:06.878 INFO:teuthology.orchestra.run.smithi060.stderr: 19.6% -- replaced with /var/log/ceph/cluster2-client.mirror.0.30318.log.gz 2024-09-22T01:02:06.878 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98556.log.gz 2024-09-22T01:02:06.878 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48966.log.gz 2024-09-22T01:02:06.878 INFO:teuthology.orchestra.run.smithi060.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109151.log.gz 2024-09-22T01:02:06.878 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.64800.log.gz 2024-09-22T01:02:06.878 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.95544.log.gz 2024-09-22T01:02:06.879 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52813.log.gz 2024-09-22T01:02:06.879 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.88984.log.gz 2024-09-22T01:02:06.879 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125066.log.gz 2024-09-22T01:02:06.879 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52927.log.gz 2024-09-22T01:02:06.879 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57674.log.gz 2024-09-22T01:02:06.879 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44928.log.gz 2024-09-22T01:02:06.879 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.119579.log.gz 2024-09-22T01:02:06.879 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.103836.log.gz 2024-09-22T01:02:06.879 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.38853.log.gz 2024-09-22T01:02:06.879 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.64016.log.gz 90.0% 2024-09-22T01:02:06.879 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.122501.log.gz 2024-09-22T01:02:06.879 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52947.log.gz 2024-09-22T01:02:06.879 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.52532.log.gz 87.4% 2024-09-22T01:02:06.879 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.106777.log.gz 2024-09-22T01:02:06.879 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109637.log.gz 2024-09-22T01:02:06.879 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.41860.log.gz 2024-09-22T01:02:06.879 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32919.log.gz 2024-09-22T01:02:06.880 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32331.log.gz 2024-09-22T01:02:06.880 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140651.log.gz 2024-09-22T01:02:06.880 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59376.log.gz 2024-09-22T01:02:06.880 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.100602.log.gz 2024-09-22T01:02:06.880 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.78694.log.gz 2024-09-22T01:02:06.880 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58934.log.gz 2024-09-22T01:02:06.880 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.113148.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.49287.log.gz 2024-09-22T01:02:06.880 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.881 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42267.log.gz 2024-09-22T01:02:06.881 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118306.log.gz 2024-09-22T01:02:06.881 INFO:teuthology.orchestra.run.smithi060.stderr: 3.5% -- replaced with /var/log/ceph/cluster2-client.mirror.30211.log.gz 2024-09-22T01:02:06.881 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93858.log.gz 2024-09-22T01:02:06.881 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57128.log.gz 2024-09-22T01:02:06.881 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.128895.log.gz 2024-09-22T01:02:06.881 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126532.log.gz 2024-09-22T01:02:06.881 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103793.log.gz 2024-09-22T01:02:06.881 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64926.log.gz 2024-09-22T01:02:06.881 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.118466.log.gz 2024-09-22T01:02:06.881 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92010.log.gz 2024-09-22T01:02:06.881 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.140906.log.gz 2024-09-22T01:02:06.881 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.63701.log.gz 2024-09-22T01:02:06.882 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113989.log.gz 2024-09-22T01:02:06.882 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60321.log.gz 2024-09-22T01:02:06.882 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44509.log.gz 2024-09-22T01:02:06.882 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.47075.log.gz 2024-09-22T01:02:06.882 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140856.log.gz 2024-09-22T01:02:06.882 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.114299.log.gz 2024-09-22T01:02:06.882 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59605.log.gz 2024-09-22T01:02:06.882 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102641.log.gz 2024-09-22T01:02:06.882 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71202.log.gz 2024-09-22T01:02:06.882 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.84389.log.gz 2024-09-22T01:02:06.882 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.70724.log.gz 2024-09-22T01:02:06.882 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84329.log.gz 2024-09-22T01:02:06.882 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35351.log.gz 2024-09-22T01:02:06.882 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.120645.log.gz 2024-09-22T01:02:06.883 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.66407.log.gz 2024-09-22T01:02:06.883 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60070.log.gz 2024-09-22T01:02:06.883 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.115709.log.gz 2024-09-22T01:02:06.883 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.87547.log.gz 2024-09-22T01:02:06.883 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81279.log.gz 2024-09-22T01:02:06.883 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.98449.log.gz 2024-09-22T01:02:06.883 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56435.log.gz 2024-09-22T01:02:06.883 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55742.log.gz 2024-09-22T01:02:06.883 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.50373.log.gz 2024-09-22T01:02:06.883 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117138.log.gz 2024-09-22T01:02:06.883 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.34789.log.gz 2024-09-22T01:02:06.883 INFO:teuthology.orchestra.run.smithi060.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.85993.log.gz 90.0% 2024-09-22T01:02:06.883 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.34748.log.gz 2024-09-22T01:02:06.883 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.107105.log.gz 2024-09-22T01:02:06.883 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.107511.log.gz 2024-09-22T01:02:06.883 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71623.log.gz 2024-09-22T01:02:06.884 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36343.log.gz 2024-09-22T01:02:06.884 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.40860.log.gz 2024-09-22T01:02:06.884 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.95611.log.gz 2024-09-22T01:02:06.884 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98237.log.gz 2024-09-22T01:02:06.884 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.53645.log.gz 2024-09-22T01:02:06.884 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.47161.log.gz 2024-09-22T01:02:06.884 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.34534.log.gz 2024-09-22T01:02:06.884 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.115876.log.gz 2024-09-22T01:02:06.884 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.86898.log.gz 2024-09-22T01:02:06.884 INFO:teuthology.orchestra.run.smithi060.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.98365.log.gz 2024-09-22T01:02:06.884 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.95772.log.gz 2024-09-22T01:02:06.884 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.120671.log.gz 2024-09-22T01:02:06.884 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.46188.log.gz 2024-09-22T01:02:06.884 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101906.log.gz 2024-09-22T01:02:06.884 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119632.log.gz 2024-09-22T01:02:06.885 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109776.log.gz 2024-09-22T01:02:06.885 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35587.log.gz 2024-09-22T01:02:06.885 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.51401.log.gz 2024-09-22T01:02:06.885 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63841.log.gz 2024-09-22T01:02:06.885 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.admin.97692.log.gz 2024-09-22T01:02:06.885 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.98428.log.gz 2024-09-22T01:02:06.885 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140501.log.gz 2024-09-22T01:02:06.885 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.104321.log.gz 2024-09-22T01:02:06.885 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.117588.log.gz 2024-09-22T01:02:06.885 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.129457.log.gz 2024-09-22T01:02:06.885 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137312.log.gz 2024-09-22T01:02:06.886 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77338.log.gz 2024-09-22T01:02:06.886 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126912.log.gz 2024-09-22T01:02:06.886 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.117696.log.gz 2024-09-22T01:02:06.886 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56962.log.gz 2024-09-22T01:02:06.886 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.138156.log.gz 90.1% 2024-09-22T01:02:06.886 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.113400.log.gz 2024-09-22T01:02:06.886 INFO:teuthology.orchestra.run.smithi060.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42237.log.gz 2024-09-22T01:02:06.886 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90540.log.gz 2024-09-22T01:02:06.886 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37792.log.gz 2024-09-22T01:02:06.886 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.98869.log.gz 2024-09-22T01:02:06.887 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68724.log.gz 2024-09-22T01:02:06.887 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.81867.log.gz 2024-09-22T01:02:06.887 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103333.log.gz 2024-09-22T01:02:06.887 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114317.log.gz 2024-09-22T01:02:06.887 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32833.log.gz 2024-09-22T01:02:06.887 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65305.log.gz 2024-09-22T01:02:06.887 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92302.log.gz 2024-09-22T01:02:06.887 INFO:teuthology.orchestra.run.smithi060.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52626.log.gz 2024-09-22T01:02:06.887 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.65540.log.gz 2024-09-22T01:02:06.887 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58976.log.gz 2024-09-22T01:02:06.887 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67674.log.gz 2024-09-22T01:02:06.887 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127541.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.81113.log.gz 2024-09-22T01:02:06.887 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.887 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123760.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.117454.log.gz 2024-09-22T01:02:06.887 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.887 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% 90.2% -- replaced with /var/log/ceph/cluster1-client.admin.120560.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.102012.log.gz 2024-09-22T01:02:06.888 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.888 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53999.log.gz 2024-09-22T01:02:06.888 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.110081.log.gz 2024-09-22T01:02:06.888 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.31435.log.gz 2024-09-22T01:02:06.888 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.97262.log.gz 2024-09-22T01:02:06.888 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114132.log.gz 2024-09-22T01:02:06.888 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.105350.log.gz 2024-09-22T01:02:06.888 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.126450.log.gz 2024-09-22T01:02:06.888 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.62054.log.gz 2024-09-22T01:02:06.888 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116549.log.gz 2024-09-22T01:02:06.888 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.118785.log.gz 2024-09-22T01:02:06.888 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123696.log.gz 2024-09-22T01:02:06.888 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.33611.log.gz 2024-09-22T01:02:06.888 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42200.log.gz 2024-09-22T01:02:06.888 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70723.log.gz 2024-09-22T01:02:06.889 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.86724.log.gz 2024-09-22T01:02:06.889 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.128718.log.gz 2024-09-22T01:02:06.889 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111464.log.gz 2024-09-22T01:02:06.889 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79306.log.gz 2024-09-22T01:02:06.889 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70426.log.gz 2024-09-22T01:02:06.889 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.95345.log.gz 2024-09-22T01:02:06.889 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66367.log.gz 2024-09-22T01:02:06.889 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122669.log.gz 2024-09-22T01:02:06.889 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124536.log.gz 2024-09-22T01:02:06.889 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70237.log.gz 2024-09-22T01:02:06.889 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132154.log.gz 2024-09-22T01:02:06.889 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.98676.log.gz 2024-09-22T01:02:06.889 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45348.log.gz 2024-09-22T01:02:06.889 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109547.log.gz 2024-09-22T01:02:06.889 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.96150.log.gz 2024-09-22T01:02:06.890 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37098.log.gz 2024-09-22T01:02:06.890 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112350.log.gz 2024-09-22T01:02:06.890 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35442.log.gz 2024-09-22T01:02:06.890 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.140174.log.gz 2024-09-22T01:02:06.890 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58387.log.gz 2024-09-22T01:02:06.890 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36931.log.gz 2024-09-22T01:02:06.890 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45390.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.81910.log.gz 2024-09-22T01:02:06.890 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.890 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127646.log.gz 2024-09-22T01:02:06.890 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104174.log.gz 2024-09-22T01:02:06.890 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.91337.log.gz 2024-09-22T01:02:06.890 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73647.log.gz 2024-09-22T01:02:06.890 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37099.log.gz 2024-09-22T01:02:06.891 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46020.log.gz 2024-09-22T01:02:06.891 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101234.log.gz 2024-09-22T01:02:06.891 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100773.log.gz 2024-09-22T01:02:06.891 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54025.log.gz 2024-09-22T01:02:06.891 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80566.log.gz 2024-09-22T01:02:06.891 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104006.log.gz 2024-09-22T01:02:06.891 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78086.log.gz 2024-09-22T01:02:06.891 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123719.log.gz 2024-09-22T01:02:06.891 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33002.log.gz 2024-09-22T01:02:06.891 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59440.log.gz 2024-09-22T01:02:06.891 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90899.log.gz 2024-09-22T01:02:06.891 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55401.log.gz 2024-09-22T01:02:06.891 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32270.log.gz 2024-09-22T01:02:06.891 INFO:teuthology.orchestra.run.smithi060.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.97303.log.gz 2024-09-22T01:02:06.892 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45349.log.gz 2024-09-22T01:02:06.892 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127793.log.gz 2024-09-22T01:02:06.892 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40090.log.gz 2024-09-22T01:02:06.892 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127858.log.gz 2024-09-22T01:02:06.892 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60762.log.gz 2024-09-22T01:02:06.892 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59902.log.gz 2024-09-22T01:02:06.892 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.95801.log.gz 2024-09-22T01:02:06.892 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.112937.log.gz 2024-09-22T01:02:06.892 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.86469.log.gz 2024-09-22T01:02:06.892 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.37602.log.gz 2024-09-22T01:02:06.892 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78926.log.gz 2024-09-22T01:02:06.892 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116633.log.gz 2024-09-22T01:02:06.892 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89573.log.gz 2024-09-22T01:02:06.893 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53269.log.gz 90.0% 2024-09-22T01:02:06.893 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.31580.log.gz 2024-09-22T01:02:06.893 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.137270.log.gz 2024-09-22T01:02:06.893 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.121685.log.gz 2024-09-22T01:02:06.893 INFO:teuthology.orchestra.run.smithi060.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87721.log.gz 2024-09-22T01:02:06.893 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90688.log.gz 2024-09-22T01:02:06.893 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36049.log.gz 2024-09-22T01:02:06.893 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93520.log.gz 2024-09-22T01:02:06.893 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.103122.log.gz 2024-09-22T01:02:06.893 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43516.log.gz 2024-09-22T01:02:06.893 INFO:teuthology.orchestra.run.smithi060.stderr: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.30126.log.gz 2024-09-22T01:02:06.893 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133950.log.gz 2024-09-22T01:02:06.893 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82077.log.gz 2024-09-22T01:02:06.893 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52512.log.gz 2024-09-22T01:02:06.894 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138323.log.gz 2024-09-22T01:02:06.894 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57465.log.gz 2024-09-22T01:02:06.894 INFO:teuthology.orchestra.run.smithi060.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.64271.log.gz 2024-09-22T01:02:06.894 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124392.log.gz 2024-09-22T01:02:06.894 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.135924.log.gz 2024-09-22T01:02:06.894 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58514.log.gz 2024-09-22T01:02:06.894 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48632.log.gz 2024-09-22T01:02:06.894 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.72018.log.gz 2024-09-22T01:02:06.894 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.50208.log.gz 2024-09-22T01:02:06.894 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102284.log.gz 2024-09-22T01:02:06.894 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60193.log.gz 2024-09-22T01:02:06.894 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66454.log.gz 2024-09-22T01:02:06.894 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114384.log.gz 2024-09-22T01:02:06.894 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34241.log.gz 2024-09-22T01:02:06.895 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67370.log.gz 2024-09-22T01:02:06.895 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.119720.log.gz 90.1% 2024-09-22T01:02:06.895 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.127711.log.gz 2024-09-22T01:02:06.895 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133367.log.gz 2024-09-22T01:02:06.895 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123802.log.gz 2024-09-22T01:02:06.895 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.39675.log.gz 2024-09-22T01:02:06.895 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.87615.log.gz 2024-09-22T01:02:06.895 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109931.log.gz 2024-09-22T01:02:06.895 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114510.log.gz 2024-09-22T01:02:06.895 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76270.log.gz 2024-09-22T01:02:06.895 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113190.log.gz 2024-09-22T01:02:06.895 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55701.log.gz 2024-09-22T01:02:06.895 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114241.log.gz 2024-09-22T01:02:06.895 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.109699.log.gz 2024-09-22T01:02:06.895 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44721.log.gz 2024-09-22T01:02:06.895 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102229.log.gz 2024-09-22T01:02:06.896 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.123592.log.gz 2024-09-22T01:02:06.896 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.37689.log.gz 2024-09-22T01:02:06.896 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102579.log.gz 2024-09-22T01:02:06.896 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.91441.log.gz 2024-09-22T01:02:06.896 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133012.log.gz 2024-09-22T01:02:06.896 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101023.log.gz 90.0% 2024-09-22T01:02:06.896 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.111284.log.gz 2024-09-22T01:02:06.896 INFO:teuthology.orchestra.run.smithi060.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.96961.log.gz 2024-09-22T01:02:06.896 INFO:teuthology.orchestra.run.smithi060.stderr: 55.8% -- replaced with /var/log/ceph/cluster2-client.mirror.30177.log.gz 2024-09-22T01:02:06.896 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79555.log.gz 2024-09-22T01:02:06.896 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64394.log.gz 90.1% 2024-09-22T01:02:06.896 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.78757.log.gz 2024-09-22T01:02:06.896 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.98529.log.gz 2024-09-22T01:02:06.896 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.49008.log.gz 2024-09-22T01:02:06.896 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70572.log.gz 2024-09-22T01:02:06.897 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78819.log.gz 2024-09-22T01:02:06.897 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59018.log.gz 2024-09-22T01:02:06.897 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123386.log.gz 2024-09-22T01:02:06.897 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133564.log.gz 2024-09-22T01:02:06.897 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53346.log.gz 2024-09-22T01:02:06.897 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.129112.log.gz 2024-09-22T01:02:06.897 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42369.log.gz 2024-09-22T01:02:06.897 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129529.log.gz 2024-09-22T01:02:06.897 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34031.log.gz 2024-09-22T01:02:06.897 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75315.log.gz 2024-09-22T01:02:06.897 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35668.log.gz 2024-09-22T01:02:06.897 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128068.log.gz 2024-09-22T01:02:06.897 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37560.log.gz 2024-09-22T01:02:06.897 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114877.log.gz 2024-09-22T01:02:06.897 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79178.log.gz 2024-09-22T01:02:06.897 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42534.log.gz 2024-09-22T01:02:06.897 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.32203.log.gz 2024-09-22T01:02:06.897 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67297.log.gz 2024-09-22T01:02:06.898 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123890.log.gz 2024-09-22T01:02:06.898 INFO:teuthology.orchestra.run.smithi060.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.97189.log.gz 2024-09-22T01:02:06.898 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78003.log.gz 2024-09-22T01:02:06.898 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45477.log.gz 2024-09-22T01:02:06.898 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78673.log.gz 2024-09-22T01:02:06.898 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.81321.log.gz 2024-09-22T01:02:06.898 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.137987.log.gz 2024-09-22T01:02:06.898 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126662.log.gz 2024-09-22T01:02:06.898 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102159.log.gz 2024-09-22T01:02:06.898 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127542.log.gz 2024-09-22T01:02:06.898 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137652.log.gz 2024-09-22T01:02:06.898 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.31618.log.gz 2024-09-22T01:02:06.898 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96567.log.gz 2024-09-22T01:02:06.898 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.77337.log.gz 2024-09-22T01:02:06.898 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55825.log.gz 2024-09-22T01:02:06.899 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124706.log.gz 2024-09-22T01:02:06.899 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105012.log.gz 2024-09-22T01:02:06.899 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103963.log.gz 2024-09-22T01:02:06.899 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.105098.log.gz 2024-09-22T01:02:06.899 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102957.log.gz 2024-09-22T01:02:06.899 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.137702.log.gz 2024-09-22T01:02:06.899 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36762.log.gz 2024-09-22T01:02:06.899 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116654.log.gz 2024-09-22T01:02:06.899 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81995.log.gz 2024-09-22T01:02:06.899 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.80817.log.gz 2024-09-22T01:02:06.899 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126985.log.gz 2024-09-22T01:02:06.899 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60488.log.gz 89.6% 2024-09-22T01:02:06.899 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.118263.log.gz 90.1% 2024-09-22T01:02:06.899 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.103208.log.gz 2024-09-22T01:02:06.900 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80440.log.gz 2024-09-22T01:02:06.900 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55276.log.gz 2024-09-22T01:02:06.900 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136259.log.gz 2024-09-22T01:02:06.900 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51065.log.gz 2024-09-22T01:02:06.900 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.74947.log.gz 2024-09-22T01:02:06.900 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.30487.log.gz 2024-09-22T01:02:06.900 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61104.log.gz 2024-09-22T01:02:06.900 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66520.log.gz 2024-09-22T01:02:06.900 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68833.log.gz 2024-09-22T01:02:06.900 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32960.log.gz 2024-09-22T01:02:06.900 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93838.log.gz 2024-09-22T01:02:06.900 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.98596.log.gz 2024-09-22T01:02:06.900 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123656.log.gz 2024-09-22T01:02:06.900 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81575.log.gz 2024-09-22T01:02:06.900 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% 90.5% -- replaced with /var/log/ceph/cluster2-client.admin.107252.log.gz 2024-09-22T01:02:06.900 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.55784.log.gz 2024-09-22T01:02:06.901 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125443.log.gz 2024-09-22T01:02:06.901 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.48335.log.gz 2024-09-22T01:02:06.901 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125344.log.gz 2024-09-22T01:02:06.901 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132239.log.gz 2024-09-22T01:02:06.901 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65073.log.gz 2024-09-22T01:02:06.901 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.106295.log.gz 2024-09-22T01:02:06.901 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116612.log.gz 2024-09-22T01:02:06.901 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37058.log.gz 2024-09-22T01:02:06.901 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56626.log.gz 2024-09-22T01:02:06.901 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102493.log.gz 2024-09-22T01:02:06.901 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88530.log.gz 2024-09-22T01:02:06.901 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.93920.log.gz 2024-09-22T01:02:06.901 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.81786.log.gz 2024-09-22T01:02:06.901 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127686.log.gz 2024-09-22T01:02:06.901 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88137.log.gz 2024-09-22T01:02:06.901 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.104508.log.gz 2024-09-22T01:02:06.902 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.admin.53129.log.gz 2024-09-22T01:02:06.902 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.31202.log.gz 90.6% 2024-09-22T01:02:06.902 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.125402.log.gz 2024-09-22T01:02:06.902 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82246.log.gz 2024-09-22T01:02:06.902 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66583.log.gz 2024-09-22T01:02:06.902 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78716.log.gz 2024-09-22T01:02:06.902 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60109.log.gz 2024-09-22T01:02:06.903 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.50209.log.gz 2024-09-22T01:02:06.903 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123931.log.gz 2024-09-22T01:02:06.903 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101024.log.gz 2024-09-22T01:02:06.903 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.96043.log.gz 2024-09-22T01:02:06.903 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.138239.log.gz 2024-09-22T01:02:06.903 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.52331.log.gz 90.0% 2024-09-22T01:02:06.904 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.37391.log.gz 2024-09-22T01:02:06.904 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45432.log.gz 2024-09-22T01:02:06.904 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68388.log.gz 2024-09-22T01:02:06.904 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38092.log.gz 2024-09-22T01:02:06.904 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.81617.log.gz 2024-09-22T01:02:06.904 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121664.log.gz 2024-09-22T01:02:06.904 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.93962.log.gz 2024-09-22T01:02:06.904 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.35878.log.gz 2024-09-22T01:02:06.904 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55615.log.gz 2024-09-22T01:02:06.904 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54720.log.gz 2024-09-22T01:02:06.905 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35272.log.gz 2024-09-22T01:02:06.905 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36761.log.gz 2024-09-22T01:02:06.905 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60741.log.gz 2024-09-22T01:02:06.905 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82035.log.gz 2024-09-22T01:02:06.905 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32050.log.gz 2024-09-22T01:02:06.905 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134117.log.gz 2024-09-22T01:02:06.905 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99433.log.gz 2024-09-22T01:02:06.905 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.109334.log.gz 2024-09-22T01:02:06.905 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.79348.log.gz 2024-09-22T01:02:06.906 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100285.log.gz 2024-09-22T01:02:06.906 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.99467.log.gz 2024-09-22T01:02:06.906 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44133.log.gz 2024-09-22T01:02:06.906 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121729.log.gz 2024-09-22T01:02:06.906 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123552.log.gz 2024-09-22T01:02:06.906 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.77142.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.82581.log.gz 2024-09-22T01:02:06.906 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.906 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100688.log.gz 2024-09-22T01:02:06.906 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.113189.log.gz 2024-09-22T01:02:06.906 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121219.log.gz 2024-09-22T01:02:06.906 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.79980.log.gz 2024-09-22T01:02:06.907 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56750.log.gz 2024-09-22T01:02:06.907 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.106585.log.gz 2024-09-22T01:02:06.907 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.82540.log.gz 2024-09-22T01:02:06.907 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.104217.log.gz 2024-09-22T01:02:06.907 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92220.log.gz 2024-09-22T01:02:06.907 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.34157.log.gz 2024-09-22T01:02:06.907 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31897.log.gz 2024-09-22T01:02:06.907 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36722.log.gz 2024-09-22T01:02:06.907 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44760.log.gz 2024-09-22T01:02:06.907 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.140863.log.gz 2024-09-22T01:02:06.907 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125862.log.gz 2024-09-22T01:02:06.907 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32497.log.gz 2024-09-22T01:02:06.907 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.84501.log.gz 2024-09-22T01:02:06.907 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.42416.log.gz 2024-09-22T01:02:06.908 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.98554.log.gz 2024-09-22T01:02:06.908 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57757.log.gz 2024-09-22T01:02:06.908 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.82415.log.gz 2024-09-22T01:02:06.908 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.37729.log.gz 2024-09-22T01:02:06.908 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.75148.log.gz 2024-09-22T01:02:06.908 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46084.log.gz 2024-09-22T01:02:06.908 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.93519.log.gz 2024-09-22T01:02:06.908 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.81195.log.gz 2024-09-22T01:02:06.908 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.admin.75289.log.gz 2024-09-22T01:02:06.908 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62422.log.gz 2024-09-22T01:02:06.908 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37730.log.gz 2024-09-22T01:02:06.908 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.128152.log.gz 2024-09-22T01:02:06.908 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46274.log.gz 2024-09-22T01:02:06.908 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.120176.log.gz 2024-09-22T01:02:06.909 INFO:teuthology.orchestra.run.smithi060.stderr: 91.4% 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.97350.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.95193.log.gz 2024-09-22T01:02:06.909 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.909 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.44172.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.95712.log.gz 2024-09-22T01:02:06.909 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.909 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.96867.log.gz 2024-09-22T01:02:06.909 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.91949.log.gz 2024-09-22T01:02:06.909 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132748.log.gz 2024-09-22T01:02:06.909 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116339.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.64069.log.gz 2024-09-22T01:02:06.909 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.909 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41498.log.gz 2024-09-22T01:02:06.909 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109256.log.gz 2024-09-22T01:02:06.909 INFO:teuthology.orchestra.run.smithi060.stderr: 56.7% -- replaced with /var/log/ceph/cluster2-client.mirror.30245.log.gz 2024-09-22T01:02:06.909 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32878.log.gz 2024-09-22T01:02:06.909 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113460.log.gz 2024-09-22T01:02:06.909 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.79768.log.gz 2024-09-22T01:02:06.910 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.admin.64157.log.gz 2024-09-22T01:02:06.910 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.102747.log.gz 2024-09-22T01:02:06.910 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.36193.log.gz 2024-09-22T01:02:06.910 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.33738.log.gz 2024-09-22T01:02:06.910 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87214.log.gz 2024-09-22T01:02:06.910 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.74833.log.gz 2024-09-22T01:02:06.910 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140686.log.gz 2024-09-22T01:02:06.910 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.95346.log.gz 2024-09-22T01:02:06.910 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33401.log.gz 2024-09-22T01:02:06.910 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105434.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.126032.log.gz 2024-09-22T01:02:06.910 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.910 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45266.log.gz 2024-09-22T01:02:06.910 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.80145.log.gz 2024-09-22T01:02:06.911 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105164.log.gz 2024-09-22T01:02:06.911 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.56434.log.gz 2024-09-22T01:02:06.911 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.118464.log.gz 2024-09-22T01:02:06.911 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127499.log.gz 2024-09-22T01:02:06.911 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.122309.log.gz 2024-09-22T01:02:06.911 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103901.log.gz 2024-09-22T01:02:06.911 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.75242.log.gz 2024-09-22T01:02:06.911 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115813.log.gz 2024-09-22T01:02:06.911 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.108749.log.gz 2024-09-22T01:02:06.911 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58641.log.gz 87.1% 2024-09-22T01:02:06.911 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.41887.log.gz 2024-09-22T01:02:06.911 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.35207.log.gz 2024-09-22T01:02:06.911 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.62742.log.gz 2024-09-22T01:02:06.911 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.128300.log.gz 2024-09-22T01:02:06.911 INFO:teuthology.orchestra.run.smithi060.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80608.log.gz 2024-09-22T01:02:06.912 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36680.log.gz 2024-09-22T01:02:06.912 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114407.log.gz 2024-09-22T01:02:06.912 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.93690.log.gz 2024-09-22T01:02:06.912 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118936.log.gz 2024-09-22T01:02:06.912 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.79765.log.gz 2024-09-22T01:02:06.912 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45013.log.gz 2024-09-22T01:02:06.912 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.94151.log.gz 2024-09-22T01:02:06.912 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34368.log.gz 2024-09-22T01:02:06.912 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80314.log.gz 2024-09-22T01:02:06.912 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68513.log.gz 2024-09-22T01:02:06.912 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64626.log.gz 2024-09-22T01:02:06.912 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.38694.log.gz 2024-09-22T01:02:06.912 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.103248.log.gz 2024-09-22T01:02:06.912 INFO:teuthology.orchestra.run.smithi060.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.119948.log.gz 2024-09-22T01:02:06.912 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.35880.log.gz 2024-09-22T01:02:06.912 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.105163.log.gz 2024-09-22T01:02:06.913 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55657.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.87999.log.gz 2024-09-22T01:02:06.913 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.913 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89572.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.43635.log.gz 2024-09-22T01:02:06.913 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.913 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109293.log.gz 2024-09-22T01:02:06.913 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77186.log.gz 2024-09-22T01:02:06.913 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.73289.log.gz 2024-09-22T01:02:06.913 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48086.log.gz 2024-09-22T01:02:06.913 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.52418.log.gz 2024-09-22T01:02:06.913 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.107251.log.gz 2024-09-22T01:02:06.913 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43392.log.gz 2024-09-22T01:02:06.913 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137525.log.gz 2024-09-22T01:02:06.913 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100393.log.gz 2024-09-22T01:02:06.913 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113544.log.gz 2024-09-22T01:02:06.913 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46443.log.gz 2024-09-22T01:02:06.914 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99425.log.gz 2024-09-22T01:02:06.914 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34998.log.gz 2024-09-22T01:02:06.914 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64842.log.gz 2024-09-22T01:02:06.914 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.39914.log.gz 2024-09-22T01:02:06.914 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89782.log.gz 2024-09-22T01:02:06.914 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34240.log.gz 2024-09-22T01:02:06.914 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37309.log.gz 2024-09-22T01:02:06.914 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.118865.log.gz 2024-09-22T01:02:06.914 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44678.log.gz 2024-09-22T01:02:06.914 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53921.log.gz 2024-09-22T01:02:06.914 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37142.log.gz 2024-09-22T01:02:06.914 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114551.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.66624.log.gz 2024-09-22T01:02:06.914 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.914 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43908.log.gz 2024-09-22T01:02:06.914 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104741.log.gz 2024-09-22T01:02:06.915 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.73538.log.gz 2024-09-22T01:02:06.915 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113567.log.gz 2024-09-22T01:02:06.915 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126426.log.gz 2024-09-22T01:02:06.915 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125612.log.gz 2024-09-22T01:02:06.915 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48714.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.37791.log.gz 2024-09-22T01:02:06.915 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.915 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.39405.log.gz 2024-09-22T01:02:06.915 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.138238.log.gz 2024-09-22T01:02:06.915 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.50948.log.gz 2024-09-22T01:02:06.915 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51863.log.gz 2024-09-22T01:02:06.915 INFO:teuthology.orchestra.run.smithi060.stderr: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.30040.log.gz 2024-09-22T01:02:06.915 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55319.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.64224.log.gz 2024-09-22T01:02:06.916 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.916 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58052.log.gz 2024-09-22T01:02:06.916 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31433.log.gz 2024-09-22T01:02:06.916 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.103962.log.gz 2024-09-22T01:02:06.916 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.95552.log.gz 2024-09-22T01:02:06.916 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46316.log.gz 2024-09-22T01:02:06.916 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.53899.log.gz 2024-09-22T01:02:06.916 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54873.log.gz 2024-09-22T01:02:06.916 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91298.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.103628.log.gz 2024-09-22T01:02:06.916 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.916 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.140756.log.gz 2024-09-22T01:02:06.916 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51651.log.gz 2024-09-22T01:02:06.916 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.64584.log.gz 2024-09-22T01:02:06.916 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.51965.log.gz 2024-09-22T01:02:06.916 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109868.log.gz 2024-09-22T01:02:06.916 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33908.log.gz 2024-09-22T01:02:06.917 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122389.log.gz 2024-09-22T01:02:06.917 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.132196.log.gz 2024-09-22T01:02:06.917 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.62747.log.gz 2024-09-22T01:02:06.917 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.34828.log.gz 2024-09-22T01:02:06.917 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.54639.log.gz 2024-09-22T01:02:06.917 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.31243.log.gz 2024-09-22T01:02:06.917 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.97645.log.gz 2024-09-22T01:02:06.917 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.64947.log.gz 2024-09-22T01:02:06.917 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46614.log.gz 2024-09-22T01:02:06.917 INFO:teuthology.orchestra.run.smithi060.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.97578.log.gz 2024-09-22T01:02:06.917 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.79935.log.gz 2024-09-22T01:02:06.918 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102786.log.gz 2024-09-22T01:02:06.918 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80407.log.gz 2024-09-22T01:02:06.918 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.121409.log.gz 2024-09-22T01:02:06.918 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.131945.log.gz 2024-09-22T01:02:06.918 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.52988.log.gz 2024-09-22T01:02:06.918 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41840.log.gz 2024-09-22T01:02:06.918 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.88330.log.gz 2024-09-22T01:02:06.918 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.120088.log.gz 2024-09-22T01:02:06.918 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.100392.log.gz 2024-09-22T01:02:06.918 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.31054.log.gz 2024-09-22T01:02:06.918 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67296.log.gz 2024-09-22T01:02:06.918 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52699.log.gz 2024-09-22T01:02:06.918 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.84371.log.gz 2024-09-22T01:02:06.918 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.102323.log.gz 2024-09-22T01:02:06.919 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101316.log.gz 2024-09-22T01:02:06.919 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.97779.log.gz 2024-09-22T01:02:06.919 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.119009.log.gz 2024-09-22T01:02:06.919 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.39615.log.gz 2024-09-22T01:02:06.919 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86408.log.gz 2024-09-22T01:02:06.919 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124013.log.gz 2024-09-22T01:02:06.919 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.64244.log.gz 90.5% 2024-09-22T01:02:06.919 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.137735.log.gz 2024-09-22T01:02:06.919 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.30822.log.gz 2024-09-22T01:02:06.919 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.51184.log.gz 2024-09-22T01:02:06.919 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41685.log.gz 2024-09-22T01:02:06.919 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.137776.log.gz 2024-09-22T01:02:06.919 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123761.log.gz 2024-09-22T01:02:06.919 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47983.log.gz 2024-09-22T01:02:06.920 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102158.log.gz 2024-09-22T01:02:06.920 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123594.log.gz 2024-09-22T01:02:06.920 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127582.log.gz 2024-09-22T01:02:06.920 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46572.log.gz 2024-09-22T01:02:06.920 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.50442.log.gz 2024-09-22T01:02:06.920 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70513.log.gz 2024-09-22T01:02:06.920 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55658.log.gz 2024-09-22T01:02:06.920 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97168.log.gz 2024-09-22T01:02:06.920 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34369.log.gz 2024-09-22T01:02:06.920 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54848.log.gz 2024-09-22T01:02:06.920 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36301.log.gz 2024-09-22T01:02:06.920 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.50439.log.gz 2024-09-22T01:02:06.920 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.82120.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.76991.log.gz 2024-09-22T01:02:06.920 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.920 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.63026.log.gz 2024-09-22T01:02:06.920 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.33698.log.gz 2024-09-22T01:02:06.921 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46613.log.gz 2024-09-22T01:02:06.921 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54319.log.gz 2024-09-22T01:02:06.921 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136032.log.gz 2024-09-22T01:02:06.921 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58261.log.gz 2024-09-22T01:02:06.921 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107089.log.gz 2024-09-22T01:02:06.921 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.81700.log.gz 2024-09-22T01:02:06.921 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.140364.log.gz 2024-09-22T01:02:06.921 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.81531.log.gz 2024-09-22T01:02:06.921 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67506.log.gz 2024-09-22T01:02:06.921 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57085.log.gz 87.3% 2024-09-22T01:02:06.922 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.74806.log.gz 2024-09-22T01:02:06.922 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109841.log.gz 2024-09-22T01:02:06.922 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45056.log.gz 2024-09-22T01:02:06.922 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45181.log.gz 2024-09-22T01:02:06.922 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.51963.log.gz 2024-09-22T01:02:06.922 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55783.log.gz 2024-09-22T01:02:06.922 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.109989.log.gz 2024-09-22T01:02:06.922 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70992.log.gz 2024-09-22T01:02:06.923 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.129115.log.gz 2024-09-22T01:02:06.923 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92218.log.gz 2024-09-22T01:02:06.923 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.100540.log.gz 2024-09-22T01:02:06.923 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54958.log.gz 2024-09-22T01:02:06.923 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.127752.log.gz 2024-09-22T01:02:06.923 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.38915.log.gz 2024-09-22T01:02:06.923 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133047.log.gz 2024-09-22T01:02:06.923 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70657.log.gz 2024-09-22T01:02:06.923 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55029.log.gz 2024-09-22T01:02:06.923 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78004.log.gz 2024-09-22T01:02:06.923 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123446.log.gz 2024-09-22T01:02:06.923 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.82373.log.gz 2024-09-22T01:02:06.923 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56834.log.gz 90.1% 2024-09-22T01:02:06.923 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.79977.log.gz 2024-09-22T01:02:06.924 INFO:teuthology.orchestra.run.smithi060.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.admin.50956.log.gz 2024-09-22T01:02:06.924 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.64338.log.gz 2024-09-22T01:02:06.924 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79597.log.gz 2024-09-22T01:02:06.924 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.52190.log.gz 90.5% 2024-09-22T01:02:06.924 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.112561.log.gz 2024-09-22T01:02:06.924 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112771.log.gz 2024-09-22T01:02:06.924 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132851.log.gz 2024-09-22T01:02:06.924 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65664.log.gz 2024-09-22T01:02:06.924 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.91299.log.gz 2024-09-22T01:02:06.924 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.80945.log.gz 2024-09-22T01:02:06.924 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.125063.log.gz 2024-09-22T01:02:06.924 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125190.log.gz 2024-09-22T01:02:06.924 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59604.log.gz 2024-09-22T01:02:06.924 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66368.log.gz 2024-09-22T01:02:06.924 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.95126.log.gz 2024-09-22T01:02:06.925 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.108220.log.gz 2024-09-22T01:02:06.925 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126953.log.gz 2024-09-22T01:02:06.925 INFO:teuthology.orchestra.run.smithi060.stderr: 68.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52284.log.gz 2024-09-22T01:02:06.925 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61202.log.gz 2024-09-22T01:02:06.925 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130487.log.gz 2024-09-22T01:02:06.925 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.36428.log.gz 2024-09-22T01:02:06.925 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114618.log.gz 2024-09-22T01:02:06.925 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45558.log.gz 2024-09-22T01:02:06.925 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60698.log.gz 2024-09-22T01:02:06.925 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77878.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.136301.log.gz 2024-09-22T01:02:06.925 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.925 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45728.log.gz 2024-09-22T01:02:06.925 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.140172.log.gz 2024-09-22T01:02:06.925 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35167.log.gz 2024-09-22T01:02:06.925 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.83968.log.gz 2024-09-22T01:02:06.926 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97148.log.gz 91.7% 2024-09-22T01:02:06.926 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.50942.log.gz 2024-09-22T01:02:06.926 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.76479.log.gz 2024-09-22T01:02:06.926 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78968.log.gz 2024-09-22T01:02:06.926 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46062.log.gz 2024-09-22T01:02:06.926 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58894.log.gz 2024-09-22T01:02:06.926 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60700.log.gz 2024-09-22T01:02:06.926 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68934.log.gz 2024-09-22T01:02:06.926 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.74692.log.gz 2024-09-22T01:02:06.926 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57925.log.gz 2024-09-22T01:02:06.926 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.100266.log.gz 2024-09-22T01:02:06.926 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43434.log.gz 2024-09-22T01:02:06.927 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54405.log.gz 2024-09-22T01:02:06.927 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45600.log.gz 2024-09-22T01:02:06.927 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.53962.log.gz 2024-09-22T01:02:06.927 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109509.log.gz 2024-09-22T01:02:06.927 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126871.log.gz 2024-09-22T01:02:06.927 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.81029.log.gz 90.0% 2024-09-22T01:02:06.927 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.125306.log.gz 2024-09-22T01:02:06.927 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.50374.log.gz 2024-09-22T01:02:06.927 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37183.log.gz 2024-09-22T01:02:06.927 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33358.log.gz 2024-09-22T01:02:06.927 INFO:teuthology.orchestra.run.smithi060.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.52901.log.gz 2024-09-22T01:02:06.927 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59652.log.gz 2024-09-22T01:02:06.927 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91109.log.gz 2024-09-22T01:02:06.928 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.65391.log.gz 2024-09-22T01:02:06.928 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37984.log.gz 2024-09-22T01:02:06.928 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.118387.log.gz 2024-09-22T01:02:06.928 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137524.log.gz 2024-09-22T01:02:06.928 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.71667.log.gz 2024-09-22T01:02:06.928 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.121453.log.gz 2024-09-22T01:02:06.928 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132995.log.gz 2024-09-22T01:02:06.928 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.61743.log.gz 2024-09-22T01:02:06.928 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105099.log.gz 2024-09-22T01:02:06.928 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44970.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.76564.log.gz 2024-09-22T01:02:06.928 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.928 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108521.log.gz 2024-09-22T01:02:06.928 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54746.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.85440.log.gz 2024-09-22T01:02:06.928 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.928 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100897.log.gz 2024-09-22T01:02:06.929 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.109446.log.gz 2024-09-22T01:02:06.929 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.132747.log.gz 2024-09-22T01:02:06.929 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34032.log.gz 2024-09-22T01:02:06.929 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% 90.0% -- replaced with /var/log/ceph/cluster2-client.admin.118018.log.gz 2024-09-22T01:02:06.929 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.80985.log.gz 2024-09-22T01:02:06.929 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.86688.log.gz 2024-09-22T01:02:06.929 INFO:teuthology.orchestra.run.smithi060.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.120290.log.gz 2024-09-22T01:02:06.929 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.77285.log.gz 2024-09-22T01:02:06.929 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110228.log.gz 2024-09-22T01:02:06.929 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104048.log.gz 2024-09-22T01:02:06.929 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37857.log.gz 2024-09-22T01:02:06.929 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125546.log.gz 2024-09-22T01:02:06.929 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.99468.log.gz 2024-09-22T01:02:06.929 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.81155.log.gz 2024-09-22T01:02:06.929 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122614.log.gz 2024-09-22T01:02:06.929 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.88895.log.gz 2024-09-22T01:02:06.930 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82799.log.gz 2024-09-22T01:02:06.930 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.48713.log.gz 2024-09-22T01:02:06.930 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116442.log.gz 2024-09-22T01:02:06.930 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91042.log.gz 2024-09-22T01:02:06.930 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132387.log.gz 2024-09-22T01:02:06.930 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.76186.log.gz 2024-09-22T01:02:06.930 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.51183.log.gz 2024-09-22T01:02:06.930 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.79177.log.gz 2024-09-22T01:02:06.930 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58136.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.137152.log.gz 2024-09-22T01:02:06.930 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.930 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123447.log.gz 2024-09-22T01:02:06.930 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112621.log.gz 2024-09-22T01:02:06.930 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70784.log.gz 2024-09-22T01:02:06.930 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.53041.log.gz 2024-09-22T01:02:06.931 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.30949.log.gz 2024-09-22T01:02:06.931 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.93562.log.gz 2024-09-22T01:02:06.931 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.132720.log.gz 2024-09-22T01:02:06.931 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.109091.log.gz 2024-09-22T01:02:06.931 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.121340.log.gz 2024-09-22T01:02:06.931 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34870.log.gz 90.6% 2024-09-22T01:02:06.931 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.56331.log.gz 2024-09-22T01:02:06.931 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77834.log.gz 2024-09-22T01:02:06.931 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.37350.log.gz 2024-09-22T01:02:06.931 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.51964.log.gz 2024-09-22T01:02:06.931 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44087.log.gz 2024-09-22T01:02:06.931 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.108267.log.gz 2024-09-22T01:02:06.931 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31729.log.gz 2024-09-22T01:02:06.931 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.36721.log.gz 2024-09-22T01:02:06.932 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49009.log.gz 2024-09-22T01:02:06.932 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81447.log.gz 2024-09-22T01:02:06.932 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112411.log.gz 2024-09-22T01:02:06.932 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102746.log.gz 2024-09-22T01:02:06.932 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100814.log.gz 2024-09-22T01:02:06.932 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116974.log.gz 2024-09-22T01:02:06.932 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88473.log.gz 2024-09-22T01:02:06.932 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.46485.log.gz 2024-09-22T01:02:06.932 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.95800.log.gz 2024-09-22T01:02:06.932 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79684.log.gz 2024-09-22T01:02:06.932 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.85698.log.gz 2024-09-22T01:02:06.932 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111282.log.gz 2024-09-22T01:02:06.932 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.133648.log.gz 2024-09-22T01:02:06.932 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.65794.log.gz 2024-09-22T01:02:06.933 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102283.log.gz 2024-09-22T01:02:06.933 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125821.log.gz 2024-09-22T01:02:06.933 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48044.log.gz 90.0% 2024-09-22T01:02:06.933 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.92134.log.gz 2024-09-22T01:02:06.933 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104952.log.gz 2024-09-22T01:02:06.933 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.84033.log.gz 2024-09-22T01:02:06.933 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102703.log.gz 2024-09-22T01:02:06.933 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46212.log.gz 2024-09-22T01:02:06.933 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68894.log.gz 2024-09-22T01:02:06.933 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.137734.log.gz 2024-09-22T01:02:06.933 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57634.log.gz 2024-09-22T01:02:06.933 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.116506.log.gz 2024-09-22T01:02:06.933 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92469.log.gz 2024-09-22T01:02:06.933 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.85034.log.gz 2024-09-22T01:02:06.933 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.92008.log.gz 2024-09-22T01:02:06.933 INFO:teuthology.orchestra.run.smithi060.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104320.log.gz 2024-09-22T01:02:06.934 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.100308.log.gz 2024-09-22T01:02:06.934 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41933.log.gz 2024-09-22T01:02:06.934 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37349.log.gz 2024-09-22T01:02:06.934 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134078.log.gz 2024-09-22T01:02:06.934 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.88605.log.gz 2024-09-22T01:02:06.934 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.76459.log.gz 2024-09-22T01:02:06.934 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.34577.log.gz 2024-09-22T01:02:06.934 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.121260.log.gz 2024-09-22T01:02:06.934 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70303.log.gz 2024-09-22T01:02:06.934 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113459.log.gz 2024-09-22T01:02:06.934 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.140454.log.gz 2024-09-22T01:02:06.935 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.108977.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.54789.log.gz 2024-09-22T01:02:06.935 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.935 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125610.log.gz 2024-09-22T01:02:06.935 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.136387.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.81280.log.gz 2024-09-22T01:02:06.935 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.935 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70698.log.gz 2024-09-22T01:02:06.935 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48462.log.gz 2024-09-22T01:02:06.935 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.102640.log.gz 2024-09-22T01:02:06.935 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.131753.log.gz 2024-09-22T01:02:06.935 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44971.log.gz 2024-09-22T01:02:06.935 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126661.log.gz 2024-09-22T01:02:06.935 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90479.log.gz 2024-09-22T01:02:06.935 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46741.log.gz 2024-09-22T01:02:06.935 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.93964.log.gz 2024-09-22T01:02:06.935 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57171.log.gz 2024-09-22T01:02:06.936 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112704.log.gz 2024-09-22T01:02:06.936 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113252.log.gz 2024-09-22T01:02:06.936 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34788.log.gz 2024-09-22T01:02:06.936 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102537.log.gz 2024-09-22T01:02:06.936 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82625.log.gz 2024-09-22T01:02:06.936 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58765.log.gz 2024-09-22T01:02:06.936 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77185.log.gz 2024-09-22T01:02:06.936 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61156.log.gz 2024-09-22T01:02:06.936 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.126805.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.56707.log.gz 2024-09-22T01:02:06.936 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.936 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32498.log.gz 2024-09-22T01:02:06.936 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67253.log.gz 2024-09-22T01:02:06.936 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56497.log.gz 2024-09-22T01:02:06.937 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45433.log.gz 2024-09-22T01:02:06.937 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78613.log.gz 2024-09-22T01:02:06.937 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43961.log.gz 2024-09-22T01:02:06.937 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82187.log.gz 2024-09-22T01:02:06.937 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.137945.log.gz 2024-09-22T01:02:06.937 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58513.log.gz 2024-09-22T01:02:06.937 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.84781.log.gz 2024-09-22T01:02:06.937 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45222.log.gz 2024-09-22T01:02:06.937 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.46529.log.gz 2024-09-22T01:02:06.937 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.39763.log.gz 2024-09-22T01:02:06.937 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.73861.log.gz 2024-09-22T01:02:06.937 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.133949.log.gz 2024-09-22T01:02:06.937 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45936.log.gz 2024-09-22T01:02:06.937 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92050.log.gz 2024-09-22T01:02:06.938 INFO:teuthology.orchestra.run.smithi060.stderr: 68.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53310.log.gz 2024-09-22T01:02:06.938 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.130063.log.gz 2024-09-22T01:02:06.938 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.72769.log.gz 90.1% 2024-09-22T01:02:06.938 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.36552.log.gz 2024-09-22T01:02:06.938 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.102116.log.gz 2024-09-22T01:02:06.938 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101800.log.gz 2024-09-22T01:02:06.938 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120925.log.gz 2024-09-22T01:02:06.938 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49477.log.gz 2024-09-22T01:02:06.938 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98006.log.gz 2024-09-22T01:02:06.938 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.46698.log.gz 2024-09-22T01:02:06.938 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65906.log.gz 2024-09-22T01:02:06.938 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.98112.log.gz 2024-09-22T01:02:06.938 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.140917.log.gz 2024-09-22T01:02:06.938 INFO:teuthology.orchestra.run.smithi060.stderr: 57.4% -- replaced with /var/log/ceph/cluster1-client.mirror.30006.log.gz 2024-09-22T01:02:06.938 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66121.log.gz 2024-09-22T01:02:06.939 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42752.log.gz 2024-09-22T01:02:06.939 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44466.log.gz 2024-09-22T01:02:06.939 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.99267.log.gz 2024-09-22T01:02:06.939 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99229.log.gz 2024-09-22T01:02:06.939 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31285.log.gz 2024-09-22T01:02:06.939 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54193.log.gz 2024-09-22T01:02:06.939 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138364.log.gz 2024-09-22T01:02:06.939 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135714.log.gz 2024-09-22T01:02:06.939 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82833.log.gz 2024-09-22T01:02:06.939 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112140.log.gz 2024-09-22T01:02:06.939 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.129885.log.gz 87.5% 2024-09-22T01:02:06.939 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.72509.log.gz 2024-09-22T01:02:06.939 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135548.log.gz 2024-09-22T01:02:06.939 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111015.log.gz 2024-09-22T01:02:06.940 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48546.log.gz 2024-09-22T01:02:06.940 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89111.log.gz 2024-09-22T01:02:06.940 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.129246.log.gz 2024-09-22T01:02:06.940 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135547.log.gz 2024-09-22T01:02:06.940 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115307.log.gz 2024-09-22T01:02:06.940 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85610.log.gz 2024-09-22T01:02:06.940 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.139772.log.gz 2024-09-22T01:02:06.940 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.98849.log.gz 2024-09-22T01:02:06.940 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83350.log.gz 2024-09-22T01:02:06.940 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.39137.log.gz 2024-09-22T01:02:06.940 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74745.log.gz 2024-09-22T01:02:06.940 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136895.log.gz 2024-09-22T01:02:06.940 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57172.log.gz 2024-09-22T01:02:06.940 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115289.log.gz 2024-09-22T01:02:06.940 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.138869.log.gz 2024-09-22T01:02:06.941 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99887.log.gz 2024-09-22T01:02:06.941 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.112874.log.gz 2024-09-22T01:02:06.941 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134328.log.gz 2024-09-22T01:02:06.941 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69187.log.gz 2024-09-22T01:02:06.941 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123174.log.gz 2024-09-22T01:02:06.941 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49093.log.gz 2024-09-22T01:02:06.941 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134999.log.gz 2024-09-22T01:02:06.941 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101802.log.gz 2024-09-22T01:02:06.941 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.38701.log.gz 2024-09-22T01:02:06.941 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.72919.log.gz 2024-09-22T01:02:06.941 INFO:teuthology.orchestra.run.smithi060.stderr: 66.4% -- replaced with /var/log/ceph/cluster1-client.admin.29938.log.gz 2024-09-22T01:02:06.941 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85493.log.gz 2024-09-22T01:02:06.941 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90244.log.gz 2024-09-22T01:02:06.941 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124116.log.gz 2024-09-22T01:02:06.941 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.34831.log.gz 2024-09-22T01:02:06.942 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115141.log.gz 2024-09-22T01:02:06.942 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69564.log.gz 2024-09-22T01:02:06.942 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100098.log.gz 2024-09-22T01:02:06.942 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44258.log.gz 2024-09-22T01:02:06.942 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55951.log.gz 2024-09-22T01:02:06.942 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93103.log.gz 2024-09-22T01:02:06.942 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110655.log.gz 2024-09-22T01:02:06.942 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69524.log.gz 2024-09-22T01:02:06.942 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.93059.log.gz 90.2% 2024-09-22T01:02:06.942 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.82983.log.gz 2024-09-22T01:02:06.942 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.46399.log.gz 2024-09-22T01:02:06.942 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134370.log.gz 2024-09-22T01:02:06.942 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.75722.log.gz 2024-09-22T01:02:06.942 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134581.log.gz 2024-09-22T01:02:06.943 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.134201.log.gz 2024-09-22T01:02:06.943 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66306.log.gz 2024-09-22T01:02:06.943 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32667.log.gz 2024-09-22T01:02:06.943 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47328.log.gz 2024-09-22T01:02:06.943 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77575.log.gz 2024-09-22T01:02:06.943 INFO:teuthology.orchestra.run.smithi060.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.119445.log.gz 2024-09-22T01:02:06.943 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111076.log.gz 2024-09-22T01:02:06.943 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136472.log.gz 2024-09-22T01:02:06.943 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109377.log.gz 2024-09-22T01:02:06.943 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139141.log.gz 2024-09-22T01:02:06.943 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56413.log.gz 2024-09-22T01:02:06.943 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92554.log.gz 2024-09-22T01:02:06.943 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66307.log.gz 2024-09-22T01:02:06.944 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.93210.log.gz 2024-09-22T01:02:06.944 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.30902.log.gz 2024-09-22T01:02:06.944 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90412.log.gz 2024-09-22T01:02:06.944 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135520.log.gz 2024-09-22T01:02:06.944 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71773.log.gz 2024-09-22T01:02:06.944 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.54998.log.gz 2024-09-22T01:02:06.944 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.41176.log.gz 2024-09-22T01:02:06.944 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92788.log.gz 2024-09-22T01:02:06.944 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.84703.log.gz 2024-09-22T01:02:06.944 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.74672.log.gz 2024-09-22T01:02:06.944 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135757.log.gz 2024-09-22T01:02:06.944 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109420.log.gz 2024-09-22T01:02:06.944 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.140910.log.gz 2024-09-22T01:02:06.944 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91625.log.gz 2024-09-22T01:02:06.944 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111357.log.gz 2024-09-22T01:02:06.945 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136598.log.gz 2024-09-22T01:02:06.945 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.122066.log.gz 2024-09-22T01:02:06.945 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41115.log.gz 2024-09-22T01:02:06.945 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.139774.log.gz 2024-09-22T01:02:06.945 INFO:teuthology.orchestra.run.smithi060.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63312.log.gz 2024-09-22T01:02:06.945 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40220.log.gz 2024-09-22T01:02:06.945 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52471.log.gz 2024-09-22T01:02:06.945 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.41404.log.gz 2024-09-22T01:02:06.945 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.73473.log.gz 2024-09-22T01:02:06.945 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105883.log.gz 2024-09-22T01:02:06.945 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46656.log.gz 2024-09-22T01:02:06.945 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.136561.log.gz 2024-09-22T01:02:06.945 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.139773.log.gz 2024-09-22T01:02:06.945 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122153.log.gz 2024-09-22T01:02:06.945 INFO:teuthology.orchestra.run.smithi060.stderr: 28.5% -- replaced with /var/log/ceph/cluster2-client.admin.25405.log.gz 2024-09-22T01:02:06.945 INFO:teuthology.orchestra.run.smithi060.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.admin.97806.log.gz 2024-09-22T01:02:06.946 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115394.log.gz 2024-09-22T01:02:06.946 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69672.log.gz 2024-09-22T01:02:06.946 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.86851.log.gz 2024-09-22T01:02:06.946 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56667.log.gz 87.6% 2024-09-22T01:02:06.946 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.141193.log.gz 2024-09-22T01:02:06.946 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92764.log.gz 2024-09-22T01:02:06.946 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.75403.log.gz 2024-09-22T01:02:06.946 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75595.log.gz 2024-09-22T01:02:06.946 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71124.log.gz 2024-09-22T01:02:06.946 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111930.log.gz 2024-09-22T01:02:06.946 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69608.log.gz 2024-09-22T01:02:06.946 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.140064.log.gz 2024-09-22T01:02:06.946 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71287.log.gz 2024-09-22T01:02:06.946 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122646.log.gz 2024-09-22T01:02:06.946 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89993.log.gz 2024-09-22T01:02:06.947 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93208.log.gz 90.1% 2024-09-22T01:02:06.947 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.124793.log.gz 2024-09-22T01:02:06.947 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.112199.log.gz 2024-09-22T01:02:06.947 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114762.log.gz 2024-09-22T01:02:06.947 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72082.log.gz 2024-09-22T01:02:06.947 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% 90.0% -- replaced with /var/log/ceph/cluster2-client.admin.28879.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.77501.log.gz 2024-09-22T01:02:06.947 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.947 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.134287.log.gz 2024-09-22T01:02:06.947 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49412.log.gz 2024-09-22T01:02:06.947 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.138785.log.gz 2024-09-22T01:02:06.947 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119310.log.gz 2024-09-22T01:02:06.947 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.93268.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.52760.log.gz 2024-09-22T01:02:06.948 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.948 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121010.log.gz 2024-09-22T01:02:06.948 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69942.log.gz 2024-09-22T01:02:06.948 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32959.log.gz 2024-09-22T01:02:06.948 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126282.log.gz 2024-09-22T01:02:06.948 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.36110.log.gz 90.0% 2024-09-22T01:02:06.948 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.137019.log.gz 2024-09-22T01:02:06.948 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115224.log.gz 2024-09-22T01:02:06.948 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.124979.log.gz 2024-09-22T01:02:06.948 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47285.log.gz 2024-09-22T01:02:06.948 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.47872.log.gz 2024-09-22T01:02:06.948 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69229.log.gz 2024-09-22T01:02:06.948 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89783.log.gz 2024-09-22T01:02:06.948 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47286.log.gz 2024-09-22T01:02:06.949 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.40947.log.gz 2024-09-22T01:02:06.949 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125022.log.gz 2024-09-22T01:02:06.949 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.83160.log.gz 2024-09-22T01:02:06.949 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74859.log.gz 2024-09-22T01:02:06.949 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59943.log.gz 2024-09-22T01:02:06.949 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115507.log.gz 2024-09-22T01:02:06.949 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.67844.log.gz 2024-09-22T01:02:06.949 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74403.log.gz 2024-09-22T01:02:06.949 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78486.log.gz 2024-09-22T01:02:06.949 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89304.log.gz 2024-09-22T01:02:06.949 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.92849.log.gz 2024-09-22T01:02:06.949 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78170.log.gz 2024-09-22T01:02:06.949 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124746.log.gz 2024-09-22T01:02:06.949 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55952.log.gz 2024-09-22T01:02:06.949 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.72503.log.gz 2024-09-22T01:02:06.950 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114972.log.gz 2024-09-22T01:02:06.950 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93004.log.gz 2024-09-22T01:02:06.950 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69356.log.gz 2024-09-22T01:02:06.950 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.77536.log.gz 2024-09-22T01:02:06.950 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.87252.log.gz 2024-09-22T01:02:06.950 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.78884.log.gz 2024-09-22T01:02:06.950 INFO:teuthology.orchestra.run.smithi060.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.108562.log.gz 2024-09-22T01:02:06.950 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.81323.log.gz 2024-09-22T01:02:06.950 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91528.log.gz 2024-09-22T01:02:06.950 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89848.log.gz 2024-09-22T01:02:06.950 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.124183.log.gz 2024-09-22T01:02:06.950 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77670.log.gz 2024-09-22T01:02:06.950 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.74578.log.gz 2024-09-22T01:02:06.950 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.137060.log.gz 2024-09-22T01:02:06.950 INFO:teuthology.orchestra.run.smithi060.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.63815.log.gz 2024-09-22T01:02:06.951 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.99633.log.gz 2024-09-22T01:02:06.951 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126176.log.gz 2024-09-22T01:02:06.951 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67927.log.gz 2024-09-22T01:02:06.951 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134411.log.gz 2024-09-22T01:02:06.951 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115508.log.gz 2024-09-22T01:02:06.951 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32582.log.gz 2024-09-22T01:02:06.951 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125547.log.gz 2024-09-22T01:02:06.951 INFO:teuthology.orchestra.run.smithi060.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.131373.log.gz 2024-09-22T01:02:06.951 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67842.log.gz 2024-09-22T01:02:06.951 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126073.log.gz 2024-09-22T01:02:06.951 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.135420.log.gz 2024-09-22T01:02:06.951 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134749.log.gz 2024-09-22T01:02:06.951 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41229.log.gz 2024-09-22T01:02:06.951 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78967.log.gz 2024-09-22T01:02:06.952 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67992.log.gz 2024-09-22T01:02:06.952 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134875.log.gz 2024-09-22T01:02:06.952 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110394.log.gz 2024-09-22T01:02:06.952 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115246.log.gz 2024-09-22T01:02:06.952 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111399.log.gz 2024-09-22T01:02:06.952 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46356.log.gz 2024-09-22T01:02:06.952 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.74350.log.gz 2024-09-22T01:02:06.952 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.38633.log.gz 2024-09-22T01:02:06.952 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49454.log.gz 2024-09-22T01:02:06.952 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.53772.log.gz 2024-09-22T01:02:06.952 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78296.log.gz 90.0% 2024-09-22T01:02:06.952 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.71076.log.gz 2024-09-22T01:02:06.952 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.41317.log.gz 2024-09-22T01:02:06.952 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70827.log.gz 2024-09-22T01:02:06.952 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116064.log.gz 2024-09-22T01:02:06.953 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126113.log.gz 2024-09-22T01:02:06.953 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.72508.log.gz 2024-09-22T01:02:06.953 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.61988.log.gz 2024-09-22T01:02:06.953 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71354.log.gz 2024-09-22T01:02:06.953 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.62049.log.gz 90.0% 2024-09-22T01:02:06.953 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.78127.log.gz 2024-09-22T01:02:06.953 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.80483.log.gz 2024-09-22T01:02:06.953 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42790.log.gz 2024-09-22T01:02:06.953 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.admin.86268.log.gz 2024-09-22T01:02:06.953 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136637.log.gz 2024-09-22T01:02:06.953 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67884.log.gz 2024-09-22T01:02:06.953 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83318.log.gz 2024-09-22T01:02:06.953 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76039.log.gz 2024-09-22T01:02:06.953 INFO:teuthology.orchestra.run.smithi060.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.31767.log.gz 2024-09-22T01:02:06.954 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.83510.log.gz 2024-09-22T01:02:06.954 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.78169.log.gz 2024-09-22T01:02:06.954 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.83794.log.gz 2024-09-22T01:02:06.954 INFO:teuthology.orchestra.run.smithi060.stderr: 79.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76755.log.gz 87.2% 2024-09-22T01:02:06.954 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.75376.log.gz 2024-09-22T01:02:06.954 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44300.log.gz 2024-09-22T01:02:06.954 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.134958.log.gz 2024-09-22T01:02:06.954 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.89824.log.gz 2024-09-22T01:02:06.954 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.129778.log.gz 2024-09-22T01:02:06.954 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90184.log.gz 2024-09-22T01:02:06.954 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75976.log.gz 2024-09-22T01:02:06.954 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.31582.log.gz 90.4% 2024-09-22T01:02:06.954 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60445.log.gz 2024-09-22T01:02:06.954 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137020.log.gz 2024-09-22T01:02:06.955 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.129113.log.gz 2024-09-22T01:02:06.955 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116021.log.gz 2024-09-22T01:02:06.955 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123321.log.gz 2024-09-22T01:02:06.955 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72227.log.gz 2024-09-22T01:02:06.955 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69732.log.gz 2024-09-22T01:02:06.955 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127079.log.gz 2024-09-22T01:02:06.955 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71243.log.gz 2024-09-22T01:02:06.955 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111130.log.gz 2024-09-22T01:02:06.955 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68598.log.gz 2024-09-22T01:02:06.955 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.132030.log.gz 2024-09-22T01:02:06.955 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124182.log.gz 2024-09-22T01:02:06.955 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.73565.log.gz 2024-09-22T01:02:06.955 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59311.log.gz 2024-09-22T01:02:06.955 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111929.log.gz 2024-09-22T01:02:06.955 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71833.log.gz 2024-09-22T01:02:06.956 INFO:teuthology.orchestra.run.smithi060.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121945.log.gz 2024-09-22T01:02:06.956 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.71352.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.86942.log.gz 2024-09-22T01:02:06.956 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.956 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.98043.log.gz 2024-09-22T01:02:06.956 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127476.log.gz 2024-09-22T01:02:06.956 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.87362.log.gz 2024-09-22T01:02:06.956 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94407.log.gz 2024-09-22T01:02:06.956 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42563.log.gz 2024-09-22T01:02:06.956 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.62751.log.gz 2024-09-22T01:02:06.956 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136680.log.gz 2024-09-22T01:02:06.956 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91612.log.gz 2024-09-22T01:02:06.956 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.134708.log.gz 2024-09-22T01:02:06.956 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44508.log.gz 2024-09-22T01:02:06.956 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.85671.log.gz 2024-09-22T01:02:06.957 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.47792.log.gz 2024-09-22T01:02:06.957 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.94469.log.gz 2024-09-22T01:02:06.957 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110611.log.gz 2024-09-22T01:02:06.957 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.109209.log.gz 2024-09-22T01:02:06.957 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84504.log.gz 2024-09-22T01:02:06.957 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115559.log.gz 2024-09-22T01:02:06.957 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42902.log.gz 2024-09-22T01:02:06.957 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.91613.log.gz 2024-09-22T01:02:06.957 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101276.log.gz 2024-09-22T01:02:06.957 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.71772.log.gz 2024-09-22T01:02:06.957 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43259.log.gz 2024-09-22T01:02:06.957 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.77716.log.gz 2024-09-22T01:02:06.957 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.77576.log.gz 2024-09-22T01:02:06.957 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124287.log.gz 2024-09-22T01:02:06.957 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134917.log.gz 2024-09-22T01:02:06.957 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.94546.log.gz 2024-09-22T01:02:06.958 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112200.log.gz 2024-09-22T01:02:06.958 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136978.log.gz 2024-09-22T01:02:06.958 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.134160.log.gz 2024-09-22T01:02:06.958 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57821.log.gz 2024-09-22T01:02:06.958 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93419.log.gz 2024-09-22T01:02:06.958 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.96385.log.gz 2024-09-22T01:02:06.958 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94353.log.gz 2024-09-22T01:02:06.958 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.117941.log.gz 2024-09-22T01:02:06.958 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94699.log.gz 2024-09-22T01:02:06.958 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.131171.log.gz 2024-09-22T01:02:06.958 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.100309.log.gz 2024-09-22T01:02:06.958 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.110947.log.gz 2024-09-22T01:02:06.958 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69313.log.gz 2024-09-22T01:02:06.958 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83206.log.gz 2024-09-22T01:02:06.958 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.95097.log.gz 2024-09-22T01:02:06.959 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92679.log.gz 2024-09-22T01:02:06.959 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.135128.log.gz 2024-09-22T01:02:06.959 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131057.log.gz 2024-09-22T01:02:06.959 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.134747.log.gz 2024-09-22T01:02:06.959 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101653.log.gz 2024-09-22T01:02:06.959 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83380.log.gz 2024-09-22T01:02:06.959 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127373.log.gz 2024-09-22T01:02:06.959 INFO:teuthology.orchestra.run.smithi060.stderr: 68.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130984.log.gz 2024-09-22T01:02:06.959 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.83432.log.gz 2024-09-22T01:02:06.959 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66732.log.gz 2024-09-22T01:02:06.959 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122815.log.gz 2024-09-22T01:02:06.959 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93143.log.gz 2024-09-22T01:02:06.959 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138742.log.gz 2024-09-22T01:02:06.959 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125021.log.gz 2024-09-22T01:02:06.960 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105742.log.gz 2024-09-22T01:02:06.960 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110432.log.gz 2024-09-22T01:02:06.960 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.119331.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.114719.log.gz 2024-09-22T01:02:06.960 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.960 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56564.log.gz 2024-09-22T01:02:06.960 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111016.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.83685.log.gz 2024-09-22T01:02:06.960 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.960 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.99230.log.gz 2024-09-22T01:02:06.960 INFO:teuthology.orchestra.run.smithi060.stderr: 68.0% 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.38907.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.130870.log.gz 2024-09-22T01:02:06.960 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.960 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.39075.log.gz 2024-09-22T01:02:06.960 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.43322.log.gz 2024-09-22T01:02:06.960 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.138668.log.gz 2024-09-22T01:02:06.960 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116087.log.gz 2024-09-22T01:02:06.960 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.75470.log.gz 2024-09-22T01:02:06.961 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.39140.log.gz 2024-09-22T01:02:06.961 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.39074.log.gz 2024-09-22T01:02:06.961 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.114341.log.gz 2024-09-22T01:02:06.961 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127246.log.gz 2024-09-22T01:02:06.961 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.63446.log.gz 2024-09-22T01:02:06.961 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.73866.log.gz 2024-09-22T01:02:06.961 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125711.log.gz 2024-09-22T01:02:06.961 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65880.log.gz 2024-09-22T01:02:06.961 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.41062.log.gz 2024-09-22T01:02:06.961 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99817.log.gz 2024-09-22T01:02:06.961 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121877.log.gz 2024-09-22T01:02:06.961 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% 89.6% -- replaced with /var/log/ceph/cluster1-client.admin.29921.log.gz 2024-09-22T01:02:06.961 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.129896.log.gz 2024-09-22T01:02:06.961 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124139.log.gz 2024-09-22T01:02:06.961 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.66792.log.gz 2024-09-22T01:02:06.962 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.127081.log.gz 2024-09-22T01:02:06.962 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.38916.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.92912.log.gz 2024-09-22T01:02:06.962 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.962 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.94811.log.gz 2024-09-22T01:02:06.962 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136721.log.gz 2024-09-22T01:02:06.962 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110734.log.gz 2024-09-22T01:02:06.962 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.44427.log.gz 2024-09-22T01:02:06.962 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74631.log.gz 2024-09-22T01:02:06.962 INFO:teuthology.orchestra.run.smithi060.stderr: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.29904.log.gz 2024-09-22T01:02:06.962 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.32834.log.gz 2024-09-22T01:02:06.962 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42985.log.gz 2024-09-22T01:02:06.962 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.83796.log.gz 2024-09-22T01:02:06.962 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134957.log.gz 2024-09-22T01:02:06.962 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115351.log.gz 2024-09-22T01:02:06.963 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.132029.log.gz 2024-09-22T01:02:06.963 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.77384.log.gz 2024-09-22T01:02:06.963 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.49289.log.gz 2024-09-22T01:02:06.963 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.89909.log.gz 2024-09-22T01:02:06.963 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101737.log.gz 2024-09-22T01:02:06.963 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.139647.log.gz 2024-09-22T01:02:06.963 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.94175.log.gz 2024-09-22T01:02:06.963 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73781.log.gz 2024-09-22T01:02:06.963 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122039.log.gz 2024-09-22T01:02:06.963 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90120.log.gz 2024-09-22T01:02:06.963 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74066.log.gz 2024-09-22T01:02:06.963 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.110828.log.gz 2024-09-22T01:02:06.963 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111225.log.gz 2024-09-22T01:02:06.963 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.102788.log.gz 2024-09-22T01:02:06.963 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139365.log.gz 2024-09-22T01:02:06.964 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69773.log.gz 2024-09-22T01:02:06.964 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.64673.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.115038.log.gz 2024-09-22T01:02:06.964 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.964 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.140915.log.gz 2024-09-22T01:02:06.964 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69398.log.gz 2024-09-22T01:02:06.964 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.139595.log.gz 2024-09-22T01:02:06.964 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.131004.log.gz 2024-09-22T01:02:06.964 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65838.log.gz 2024-09-22T01:02:06.964 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101736.log.gz 2024-09-22T01:02:06.964 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124326.log.gz 2024-09-22T01:02:06.964 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.79474.log.gz 2024-09-22T01:02:06.964 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44214.log.gz 2024-09-22T01:02:06.964 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82877.log.gz 2024-09-22T01:02:06.964 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33318.log.gz 2024-09-22T01:02:06.964 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100057.log.gz 2024-09-22T01:02:06.965 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.43154.log.gz 2024-09-22T01:02:06.965 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97832.log.gz 2024-09-22T01:02:06.965 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47371.log.gz 2024-09-22T01:02:06.965 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69104.log.gz 2024-09-22T01:02:06.965 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137355.log.gz 2024-09-22T01:02:06.965 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.75758.log.gz 2024-09-22T01:02:06.965 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122021.log.gz 2024-09-22T01:02:06.965 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69461.log.gz 2024-09-22T01:02:06.965 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42963.log.gz 2024-09-22T01:02:06.965 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.89950.log.gz 2024-09-22T01:02:06.965 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66918.log.gz 2024-09-22T01:02:06.965 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59061.log.gz 2024-09-22T01:02:06.965 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.40352.log.gz 2024-09-22T01:02:06.965 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.138322.log.gz 2024-09-22T01:02:06.965 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111017.log.gz 2024-09-22T01:02:06.966 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.75631.log.gz 2024-09-22T01:02:06.966 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99166.log.gz 2024-09-22T01:02:06.966 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34327.log.gz 2024-09-22T01:02:06.966 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83267.log.gz 2024-09-22T01:02:06.966 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.33211.log.gz 2024-09-22T01:02:06.966 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127162.log.gz 2024-09-22T01:02:06.966 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59944.log.gz 2024-09-22T01:02:06.966 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.122856.log.gz 2024-09-22T01:02:06.966 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.82835.log.gz 2024-09-22T01:02:06.966 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134243.log.gz 2024-09-22T01:02:06.966 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122711.log.gz 2024-09-22T01:02:06.966 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.105980.log.gz 2024-09-22T01:02:06.966 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126240.log.gz 2024-09-22T01:02:06.966 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47685.log.gz 2024-09-22T01:02:06.967 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121919.log.gz 2024-09-22T01:02:06.967 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.40359.log.gz 2024-09-22T01:02:06.967 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59857.log.gz 2024-09-22T01:02:06.967 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136934.log.gz 2024-09-22T01:02:06.967 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67926.log.gz 2024-09-22T01:02:06.967 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47832.log.gz 2024-09-22T01:02:06.967 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.37987.log.gz 2024-09-22T01:02:06.967 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.39136.log.gz 2024-09-22T01:02:06.967 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126281.log.gz 2024-09-22T01:02:06.967 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124349.log.gz 2024-09-22T01:02:06.967 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116484.log.gz 2024-09-22T01:02:06.967 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.124707.log.gz 2024-09-22T01:02:06.967 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119290.log.gz 2024-09-22T01:02:06.967 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89531.log.gz 2024-09-22T01:02:06.968 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57926.log.gz 2024-09-22T01:02:06.968 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.88790.log.gz 2024-09-22T01:02:06.968 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69144.log.gz 2024-09-22T01:02:06.968 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.115601.log.gz 2024-09-22T01:02:06.968 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.73508.log.gz 2024-09-22T01:02:06.968 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93420.log.gz 2024-09-22T01:02:06.968 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.78423.log.gz 2024-09-22T01:02:06.968 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110697.log.gz 2024-09-22T01:02:06.968 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116131.log.gz 2024-09-22T01:02:06.968 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122306.log.gz 2024-09-22T01:02:06.968 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89428.log.gz 2024-09-22T01:02:06.968 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77628.log.gz 2024-09-22T01:02:06.968 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.78379.log.gz 2024-09-22T01:02:06.968 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138996.log.gz 2024-09-22T01:02:06.968 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100204.log.gz 90.5% 2024-09-22T01:02:06.969 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.71353.log.gz 2024-09-22T01:02:06.969 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124225.log.gz 2024-09-22T01:02:06.969 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42668.log.gz 2024-09-22T01:02:06.969 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41270.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.137061.log.gz 2024-09-22T01:02:06.969 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.969 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.130890.log.gz 2024-09-22T01:02:06.969 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65981.log.gz 2024-09-22T01:02:06.969 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116086.log.gz 2024-09-22T01:02:06.969 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89153.log.gz 2024-09-22T01:02:06.969 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.admin.128703.log.gz 2024-09-22T01:02:06.969 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37393.log.gz 2024-09-22T01:02:06.969 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.93099.log.gz 2024-09-22T01:02:06.969 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69563.log.gz 2024-09-22T01:02:06.969 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.131685.log.gz 2024-09-22T01:02:06.969 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69044.log.gz 2024-09-22T01:02:06.969 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.131285.log.gz 2024-09-22T01:02:06.970 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.78378.log.gz 2024-09-22T01:02:06.970 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89303.log.gz 2024-09-22T01:02:06.970 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71122.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.91738.log.gz 2024-09-22T01:02:06.970 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.970 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.83686.log.gz 2024-09-22T01:02:06.970 INFO:teuthology.orchestra.run.smithi060.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.131440.log.gz 2024-09-22T01:02:06.970 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112097.log.gz 2024-09-22T01:02:06.970 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44382.log.gz 2024-09-22T01:02:06.970 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.135587.log.gz 2024-09-22T01:02:06.970 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42689.log.gz 2024-09-22T01:02:06.970 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71477.log.gz 2024-09-22T01:02:06.970 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93184.log.gz 2024-09-22T01:02:06.970 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135041.log.gz 2024-09-22T01:02:06.970 INFO:teuthology.orchestra.run.smithi060.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.97873.log.gz 2024-09-22T01:02:06.970 INFO:teuthology.orchestra.run.smithi060.stderr: 68.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130756.log.gz 2024-09-22T01:02:06.971 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134159.log.gz 2024-09-22T01:02:06.971 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.130662.log.gz 2024-09-22T01:02:06.971 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126804.log.gz 2024-09-22T01:02:06.971 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.131687.log.gz 2024-09-22T01:02:06.971 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.131643.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.111075.log.gz 2024-09-22T01:02:06.971 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.971 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69986.log.gz 2024-09-22T01:02:06.971 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.67885.log.gz 2024-09-22T01:02:06.971 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90454.log.gz 2024-09-22T01:02:06.971 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.80736.log.gz 2024-09-22T01:02:06.971 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44551.log.gz 2024-09-22T01:02:06.971 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.98491.log.gz 2024-09-22T01:02:06.972 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68305.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.81618.log.gz 2024-09-22T01:02:06.972 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.972 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92158.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.77715.log.gz 2024-09-22T01:02:06.972 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.972 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133991.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.111822.log.gz 2024-09-22T01:02:06.972 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.972 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79054.log.gz 2024-09-22T01:02:06.972 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56498.log.gz 2024-09-22T01:02:06.972 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68136.log.gz 2024-09-22T01:02:06.972 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138448.log.gz 2024-09-22T01:02:06.972 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.137986.log.gz 2024-09-22T01:02:06.972 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36470.log.gz 2024-09-22T01:02:06.972 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.79136.log.gz 2024-09-22T01:02:06.972 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67338.log.gz 2024-09-22T01:02:06.973 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.83552.log.gz 2024-09-22T01:02:06.973 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46783.log.gz 2024-09-22T01:02:06.973 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75087.log.gz 2024-09-22T01:02:06.973 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110396.log.gz 2024-09-22T01:02:06.973 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43197.log.gz 2024-09-22T01:02:06.973 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91442.log.gz 2024-09-22T01:02:06.973 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90270.log.gz 2024-09-22T01:02:06.973 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.135631.log.gz 2024-09-22T01:02:06.973 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.63218.log.gz 2024-09-22T01:02:06.973 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136808.log.gz 2024-09-22T01:02:06.973 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113082.log.gz 2024-09-22T01:02:06.973 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.78485.log.gz 2024-09-22T01:02:06.973 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.55402.log.gz 2024-09-22T01:02:06.973 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.81237.log.gz 89.7% 2024-09-22T01:02:06.973 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.72873.log.gz 2024-09-22T01:02:06.973 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.95060.log.gz 90.1% 2024-09-22T01:02:06.974 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.138975.log.gz 2024-09-22T01:02:06.974 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135338.log.gz 2024-09-22T01:02:06.974 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138826.log.gz 2024-09-22T01:02:06.974 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135295.log.gz 2024-09-22T01:02:06.974 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103837.log.gz 2024-09-22T01:02:06.974 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.106401.log.gz 2024-09-22T01:02:06.974 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33043.log.gz 2024-09-22T01:02:06.974 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.131399.log.gz 2024-09-22T01:02:06.974 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37310.log.gz 2024-09-22T01:02:06.974 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.35629.log.gz 2024-09-22T01:02:06.974 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59522.log.gz 2024-09-22T01:02:06.974 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.34092.log.gz 2024-09-22T01:02:06.974 INFO:teuthology.orchestra.run.smithi060.stderr: 86.5% -- replaced with /var/log/ceph/cluster1-client.admin.139593.log.gz 2024-09-22T01:02:06.974 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.75673.log.gz 2024-09-22T01:02:06.974 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.141192.log.gz 2024-09-22T01:02:06.975 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89193.log.gz 2024-09-22T01:02:06.975 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136638.log.gz 2024-09-22T01:02:06.975 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.93418.log.gz 2024-09-22T01:02:06.975 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.admin.131259.log.gz 2024-09-22T01:02:06.975 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.85879.log.gz 2024-09-22T01:02:06.975 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115143.log.gz 2024-09-22T01:02:06.975 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136850.log.gz 2024-09-22T01:02:06.975 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.112031.log.gz 2024-09-22T01:02:06.975 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.73028.log.gz 2024-09-22T01:02:06.975 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116022.log.gz 2024-09-22T01:02:06.975 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49679.log.gz 2024-09-22T01:02:06.975 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123299.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.59102.log.gz 2024-09-22T01:02:06.975 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.975 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59355.log.gz 2024-09-22T01:02:06.975 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.72421.log.gz 2024-09-22T01:02:06.976 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44340.log.gz 2024-09-22T01:02:06.976 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123172.log.gz 2024-09-22T01:02:06.976 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47495.log.gz 2024-09-22T01:02:06.976 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126177.log.gz 2024-09-22T01:02:06.976 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48587.log.gz 2024-09-22T01:02:06.976 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122963.log.gz 2024-09-22T01:02:06.976 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44173.log.gz 2024-09-22T01:02:06.976 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77627.log.gz 2024-09-22T01:02:06.976 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.77492.log.gz 2024-09-22T01:02:06.976 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.100140.log.gz 2024-09-22T01:02:06.976 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.72947.log.gz 2024-09-22T01:02:06.976 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.127122.log.gz 2024-09-22T01:02:06.977 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111864.log.gz 2024-09-22T01:02:06.977 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136807.log.gz 2024-09-22T01:02:06.977 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76159.log.gz 2024-09-22T01:02:06.977 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.135419.log.gz 2024-09-22T01:02:06.977 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124074.log.gz 2024-09-22T01:02:06.977 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.83797.log.gz 2024-09-22T01:02:06.977 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135167.log.gz 2024-09-22T01:02:06.977 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135652.log.gz 2024-09-22T01:02:06.977 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.98148.log.gz 2024-09-22T01:02:06.977 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71125.log.gz 2024-09-22T01:02:06.977 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.116296.log.gz 2024-09-22T01:02:06.977 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91569.log.gz 2024-09-22T01:02:06.977 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.47454.log.gz 2024-09-22T01:02:06.977 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60487.log.gz 2024-09-22T01:02:06.978 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100182.log.gz 2024-09-22T01:02:06.978 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.83793.log.gz 2024-09-22T01:02:06.978 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43132.log.gz 2024-09-22T01:02:06.978 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.admin.129014.log.gz 2024-09-22T01:02:06.978 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.83509.log.gz 2024-09-22T01:02:06.978 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.43006.log.gz 2024-09-22T01:02:06.978 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105899.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.127332.log.gz 2024-09-22T01:02:06.978 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.978 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66943.log.gz 90.1% 2024-09-22T01:02:06.978 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.110696.log.gz 2024-09-22T01:02:06.978 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.123213.log.gz 2024-09-22T01:02:06.978 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139173.log.gz 2024-09-22T01:02:06.978 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.63332.log.gz 2024-09-22T01:02:06.978 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90268.log.gz 2024-09-22T01:02:06.979 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.122965.log.gz 2024-09-22T01:02:06.979 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124916.log.gz 2024-09-22T01:02:06.979 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136511.log.gz 2024-09-22T01:02:06.979 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.72993.log.gz 2024-09-22T01:02:06.979 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.101864.log.gz 2024-09-22T01:02:06.979 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.32090.log.gz 2024-09-22T01:02:06.979 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.74464.log.gz 2024-09-22T01:02:06.979 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44844.log.gz 2024-09-22T01:02:06.979 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127163.log.gz 2024-09-22T01:02:06.979 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135379.log.gz 2024-09-22T01:02:06.979 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59228.log.gz 2024-09-22T01:02:06.979 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.134412.log.gz 2024-09-22T01:02:06.979 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115099.log.gz 2024-09-22T01:02:06.979 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.93269.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.98023.log.gz 2024-09-22T01:02:06.979 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.980 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.130829.log.gz 2024-09-22T01:02:06.980 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78338.log.gz 2024-09-22T01:02:06.980 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90118.log.gz 2024-09-22T01:02:06.980 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.134665.log.gz 2024-09-22T01:02:06.980 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42837.log.gz 2024-09-22T01:02:06.980 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.49329.log.gz 2024-09-22T01:02:06.980 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93309.log.gz 2024-09-22T01:02:06.980 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.134244.log.gz 2024-09-22T01:02:06.980 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110950.log.gz 2024-09-22T01:02:06.980 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71813.log.gz 2024-09-22T01:02:06.980 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69438.log.gz 2024-09-22T01:02:06.980 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44256.log.gz 2024-09-22T01:02:06.980 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.78214.log.gz 2024-09-22T01:02:06.980 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124956.log.gz 2024-09-22T01:02:06.981 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.83431.log.gz 90.1% 2024-09-22T01:02:06.981 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59858.log.gz 2024-09-22T01:02:06.981 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72050.log.gz 2024-09-22T01:02:06.981 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79683.log.gz 2024-09-22T01:02:06.981 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33087.log.gz 2024-09-22T01:02:06.981 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.50951.log.gz 2024-09-22T01:02:06.981 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108997.log.gz 2024-09-22T01:02:06.981 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116192.log.gz 2024-09-22T01:02:06.981 INFO:teuthology.orchestra.run.smithi060.stderr: 68.3% 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.74558.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.122879.log.gz 2024-09-22T01:02:06.981 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.981 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66163.log.gz 2024-09-22T01:02:06.981 INFO:teuthology.orchestra.run.smithi060.stderr: 68.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131098.log.gz 2024-09-22T01:02:06.981 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67783.log.gz 2024-09-22T01:02:06.981 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.121983.log.gz 2024-09-22T01:02:06.981 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33169.log.gz 2024-09-22T01:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123026.log.gz 2024-09-22T01:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59717.log.gz 2024-09-22T01:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47686.log.gz 2024-09-22T01:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139219.log.gz 2024-09-22T01:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47687.log.gz 2024-09-22T01:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124030.log.gz 2024-09-22T01:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77669.log.gz 2024-09-22T01:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.134621.log.gz 2024-09-22T01:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83265.log.gz 2024-09-22T01:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138912.log.gz 2024-09-22T01:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111890.log.gz 2024-09-22T01:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.75849.log.gz 2024-09-22T01:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.92580.log.gz 2024-09-22T01:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77383.log.gz 2024-09-22T01:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.80461.log.gz 2024-09-22T01:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101862.log.gz 2024-09-22T01:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134557.log.gz 2024-09-22T01:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.47748.log.gz 2024-09-22T01:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.130803.log.gz 2024-09-22T01:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.131232.log.gz 2024-09-22T01:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135168.log.gz 2024-09-22T01:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69460.log.gz 2024-09-22T01:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119404.log.gz 2024-09-22T01:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122712.log.gz 2024-09-22T01:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59185.log.gz 2024-09-22T01:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44383.log.gz 2024-09-22T01:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89363.log.gz 2024-09-22T01:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136679.log.gz 2024-09-22T01:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69043.log.gz 2024-09-22T01:02:06.984 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123027.log.gz 2024-09-22T01:02:06.984 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71328.log.gz 2024-09-22T01:02:06.984 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111613.log.gz 2024-09-22T01:02:06.984 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89279.log.gz 2024-09-22T01:02:06.984 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77535.log.gz 2024-09-22T01:02:06.984 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.70581.log.gz 2024-09-22T01:02:06.984 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87361.log.gz 2024-09-22T01:02:06.984 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59377.log.gz 2024-09-22T01:02:06.984 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.101106.log.gz 2024-09-22T01:02:06.984 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75720.log.gz 2024-09-22T01:02:06.984 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.37453.log.gz 2024-09-22T01:02:06.984 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47413.log.gz 2024-09-22T01:02:06.984 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124932.log.gz 2024-09-22T01:02:06.984 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.101696.log.gz 2024-09-22T01:02:06.984 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99332.log.gz 2024-09-22T01:02:06.985 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.47162.log.gz 2024-09-22T01:02:06.985 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111612.log.gz 2024-09-22T01:02:06.985 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.101358.log.gz 2024-09-22T01:02:06.985 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126216.log.gz 2024-09-22T01:02:06.985 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% 90.2% -- replaced with /var/log/ceph/cluster1-client.admin.130776.log.gz 2024-09-22T01:02:06.985 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.36823.log.gz 2024-09-22T01:02:06.985 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37991.log.gz 2024-09-22T01:02:06.985 INFO:teuthology.orchestra.run.smithi060.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109132.log.gz 2024-09-22T01:02:06.985 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43962.log.gz 2024-09-22T01:02:06.985 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60531.log.gz 87.7% 2024-09-22T01:02:06.985 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.99187.log.gz 2024-09-22T01:02:06.985 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71413.log.gz 2024-09-22T01:02:06.985 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133658.log.gz 2024-09-22T01:02:06.985 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.33044.log.gz 2024-09-22T01:02:06.986 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.95897.log.gz 2024-09-22T01:02:06.986 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126386.log.gz 2024-09-22T01:02:06.986 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66006.log.gz 2024-09-22T01:02:06.986 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111721.log.gz 2024-09-22T01:02:06.986 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33210.log.gz 2024-09-22T01:02:06.986 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.88279.log.gz 2024-09-22T01:02:06.986 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.89404.log.gz 2024-09-22T01:02:06.986 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.79093.log.gz 2024-09-22T01:02:06.986 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43234.log.gz 2024-09-22T01:02:06.986 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.86382.log.gz 2024-09-22T01:02:06.986 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.100896.log.gz 2024-09-22T01:02:06.986 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.134453.log.gz 2024-09-22T01:02:06.986 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.133868.log.gz 2024-09-22T01:02:06.986 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112074.log.gz 2024-09-22T01:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.137904.log.gz 2024-09-22T01:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43839.log.gz 2024-09-22T01:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.71476.log.gz 2024-09-22T01:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126746.log.gz 2024-09-22T01:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44301.log.gz 2024-09-22T01:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.49499.log.gz 2024-09-22T01:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71624.log.gz 2024-09-22T01:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.130347.log.gz 2024-09-22T01:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135127.log.gz 90.4% 2024-09-22T01:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.69858.log.gz 2024-09-22T01:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101570.log.gz 2024-09-22T01:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.39961.log.gz 2024-09-22T01:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56203.log.gz 2024-09-22T01:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.100938.log.gz 2024-09-22T01:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43260.log.gz 2024-09-22T01:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.91211.log.gz 2024-09-22T01:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.37601.log.gz 2024-09-22T01:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82698.log.gz 2024-09-22T01:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.39866.log.gz 2024-09-22T01:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59815.log.gz 2024-09-22T01:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35440.log.gz 2024-09-22T01:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92578.log.gz 2024-09-22T01:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115854.log.gz 2024-09-22T01:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136763.log.gz 2024-09-22T01:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.112560.log.gz 2024-09-22T01:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111677.log.gz 2024-09-22T01:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92051.log.gz 2024-09-22T01:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137668.log.gz 2024-09-22T01:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105504.log.gz 2024-09-22T01:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114343.log.gz 2024-09-22T01:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55993.log.gz 2024-09-22T01:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122752.log.gz 2024-09-22T01:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32269.log.gz 2024-09-22T01:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36258.log.gz 2024-09-22T01:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.118713.log.gz 2024-09-22T01:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123866.log.gz 2024-09-22T01:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.138827.log.gz 2024-09-22T01:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.107517.log.gz 2024-09-22T01:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110291.log.gz 2024-09-22T01:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111181.log.gz 2024-09-22T01:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.120222.log.gz 2024-09-22T01:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115919.log.gz 2024-09-22T01:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131305.log.gz 2024-09-22T01:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.116514.log.gz 2024-09-22T01:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32049.log.gz 2024-09-22T01:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.34118.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.107621.log.gz 2024-09-22T01:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.74444.log.gz 2024-09-22T01:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45960.log.gz 2024-09-22T01:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.31945.log.gz 2024-09-22T01:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.129276.log.gz 2024-09-22T01:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102367.log.gz 2024-09-22T01:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37059.log.gz 2024-09-22T01:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67632.log.gz 2024-09-22T01:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.97236.log.gz 2024-09-22T01:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121836.log.gz 2024-09-22T01:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59773.log.gz 2024-09-22T01:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.120336.log.gz 2024-09-22T01:02:06.991 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.130461.log.gz 2024-09-22T01:02:06.991 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.90058.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.71898.log.gz 2024-09-22T01:02:06.991 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.991 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124222.log.gz 2024-09-22T01:02:06.991 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.117159.log.gz 2024-09-22T01:02:06.991 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75974.log.gz 2024-09-22T01:02:06.991 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54258.log.gz 2024-09-22T01:02:06.991 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63727.log.gz 2024-09-22T01:02:06.991 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89639.log.gz 2024-09-22T01:02:06.991 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123322.log.gz 2024-09-22T01:02:06.991 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91443.log.gz 2024-09-22T01:02:06.991 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67004.log.gz 2024-09-22T01:02:06.991 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.51671.log.gz 2024-09-22T01:02:06.991 INFO:teuthology.orchestra.run.smithi060.stderr: 80.5% -- replaced with /var/log/ceph/cluster1-client.mirror.30604.log.gz 2024-09-22T01:02:06.992 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76418.log.gz 2024-09-22T01:02:06.992 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.109594.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.37989.log.gz 2024-09-22T01:02:06.992 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.992 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134371.log.gz 2024-09-22T01:02:06.992 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41384.log.gz 2024-09-22T01:02:06.992 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44130.log.gz 2024-09-22T01:02:06.992 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43196.log.gz 2024-09-22T01:02:06.992 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91570.log.gz 2024-09-22T01:02:06.992 INFO:teuthology.orchestra.run.smithi060.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.76416.log.gz 2024-09-22T01:02:06.992 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59774.log.gz 2024-09-22T01:02:06.992 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110461.log.gz 2024-09-22T01:02:06.992 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.107752.log.gz 2024-09-22T01:02:06.992 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.107750.log.gz 2024-09-22T01:02:06.992 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.132342.log.gz 2024-09-22T01:02:06.992 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.98702.log.gz 2024-09-22T01:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.73677.log.gz 2024-09-22T01:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.76522.log.gz 2024-09-22T01:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.50443.log.gz 2024-09-22T01:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.84483.log.gz 2024-09-22T01:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.62591.log.gz 2024-09-22T01:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.77011.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.108065.log.gz 2024-09-22T01:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.108407.log.gz 2024-09-22T01:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.86599.log.gz 2024-09-22T01:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.76602.log.gz 2024-09-22T01:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.121637.log.gz 2024-09-22T01:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107119.log.gz 2024-09-22T01:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45140.log.gz 2024-09-22T01:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134705.log.gz 2024-09-22T01:02:06.994 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67371.log.gz 2024-09-22T01:02:06.994 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44886.log.gz 2024-09-22T01:02:06.994 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.70363.log.gz 2024-09-22T01:02:06.994 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45516.log.gz 2024-09-22T01:02:06.994 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.123132.log.gz 2024-09-22T01:02:06.994 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87279.log.gz 2024-09-22T01:02:06.994 INFO:teuthology.orchestra.run.smithi060.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110142.log.gz 2024-09-22T01:02:06.994 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37033.log.gz 2024-09-22T01:02:06.994 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32271.log.gz 2024-09-22T01:02:06.994 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112727.log.gz 2024-09-22T01:02:06.994 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122543.log.gz 2024-09-22T01:02:06.994 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92259.log.gz 2024-09-22T01:02:06.994 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135881.log.gz 2024-09-22T01:02:06.994 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57548.log.gz 2024-09-22T01:02:06.994 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46868.log.gz 2024-09-22T01:02:06.995 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126702.log.gz 2024-09-22T01:02:06.995 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136091.log.gz 2024-09-22T01:02:06.995 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135000.log.gz 2024-09-22T01:02:06.995 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123973.log.gz 2024-09-22T01:02:06.995 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55912.log.gz 2024-09-22T01:02:06.995 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.82245.log.gz 2024-09-22T01:02:06.995 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81198.log.gz 2024-09-22T01:02:06.995 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127709.log.gz 2024-09-22T01:02:06.995 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70722.log.gz 2024-09-22T01:02:06.995 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.81111.log.gz 2024-09-22T01:02:06.995 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59816.log.gz 2024-09-22T01:02:06.995 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43112.log.gz 2024-09-22T01:02:06.995 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138868.log.gz 2024-09-22T01:02:06.995 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.130917.log.gz 2024-09-22T01:02:06.995 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.39359.log.gz 2024-09-22T01:02:06.996 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.48503.log.gz 2024-09-22T01:02:06.996 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.114197.log.gz 2024-09-22T01:02:06.996 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72288.log.gz 2024-09-22T01:02:06.996 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.136935.log.gz 2024-09-22T01:02:06.996 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37519.log.gz 2024-09-22T01:02:06.996 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94545.log.gz 2024-09-22T01:02:06.996 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.79895.log.gz 2024-09-22T01:02:06.996 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.61706.log.gz 2024-09-22T01:02:06.996 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33275.log.gz 2024-09-22T01:02:06.996 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102221.log.gz 2024-09-22T01:02:06.996 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.87572.log.gz 2024-09-22T01:02:06.996 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138932.log.gz 2024-09-22T01:02:06.996 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69884.log.gz 2024-09-22T01:02:06.996 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.106726.log.gz 2024-09-22T01:02:06.996 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.119606.log.gz 2024-09-22T01:02:06.997 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92999.log.gz 2024-09-22T01:02:06.997 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.136764.log.gz 2024-09-22T01:02:06.997 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.45643.log.gz 91.5% 2024-09-22T01:02:06.997 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.52559.log.gz 2024-09-22T01:02:06.997 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58389.log.gz 2024-09-22T01:02:06.997 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.48377.log.gz 2024-09-22T01:02:06.997 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80901.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.43476.log.gz 2024-09-22T01:02:06.997 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.997 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.admin.106905.log.gz 2024-09-22T01:02:06.997 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.33128.log.gz 2024-09-22T01:02:06.997 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.134497.log.gz 2024-09-22T01:02:06.997 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60655.log.gz 2024-09-22T01:02:06.997 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.89929.log.gz 2024-09-22T01:02:06.997 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.106687.log.gz 2024-09-22T01:02:06.997 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.140914.log.gz 2024-09-22T01:02:06.998 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111990.log.gz 2024-09-22T01:02:06.998 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.128997.log.gz 2024-09-22T01:02:06.998 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136512.log.gz 2024-09-22T01:02:06.998 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80650.log.gz 2024-09-22T01:02:06.998 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103166.log.gz 2024-09-22T01:02:06.998 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42900.log.gz 2024-09-22T01:02:06.998 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104886.log.gz 2024-09-22T01:02:06.998 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59144.log.gz 2024-09-22T01:02:06.998 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56625.log.gz 91.7% 2024-09-22T01:02:06.998 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.107440.log.gz 2024-09-22T01:02:06.998 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92638.log.gz 2024-09-22T01:02:06.998 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77833.log.gz 2024-09-22T01:02:06.998 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.83798.log.gz 2024-09-22T01:02:06.998 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.95078.log.gz 2024-09-22T01:02:06.998 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.4% 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.115980.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.104594.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.106943.log.gz 2024-09-22T01:02:06.999 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.999 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.999 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.128070.log.gz 2024-09-22T01:02:06.999 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.71454.log.gz 2024-09-22T01:02:06.999 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71706.log.gz 2024-09-22T01:02:06.999 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111720.log.gz 2024-09-22T01:02:06.999 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.134118.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.115878.log.gz 2024-09-22T01:02:06.999 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:06.999 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138492.log.gz 2024-09-22T01:02:06.999 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33129.log.gz 2024-09-22T01:02:06.999 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71748.log.gz 2024-09-22T01:02:06.999 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137608.log.gz 2024-09-22T01:02:06.999 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124286.log.gz 2024-09-22T01:02:06.999 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79264.log.gz 2024-09-22T01:02:06.999 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.112096.log.gz 2024-09-22T01:02:07.000 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122882.log.gz 2024-09-22T01:02:07.000 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.80859.log.gz 2024-09-22T01:02:07.000 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79513.log.gz 2024-09-22T01:02:07.000 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80776.log.gz 2024-09-22T01:02:07.000 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78043.log.gz 2024-09-22T01:02:07.000 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138493.log.gz 2024-09-22T01:02:07.000 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58642.log.gz 2024-09-22T01:02:07.000 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33253.log.gz 2024-09-22T01:02:07.000 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.123236.log.gz 2024-09-22T01:02:07.000 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.136894.log.gz 2024-09-22T01:02:07.000 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115769.log.gz 90.0% 2024-09-22T01:02:07.000 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.57382.log.gz 2024-09-22T01:02:07.000 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32228.log.gz 2024-09-22T01:02:07.000 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.35501.log.gz 2024-09-22T01:02:07.001 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.139894.log.gz 2024-09-22T01:02:07.001 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.admin.50761.log.gz 2024-09-22T01:02:07.001 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60656.log.gz 2024-09-22T01:02:07.001 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.128708.log.gz 2024-09-22T01:02:07.001 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.37559.log.gz 2024-09-22T01:02:07.001 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.133137.log.gz 2024-09-22T01:02:07.001 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78925.log.gz 2024-09-22T01:02:07.001 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59901.log.gz 2024-09-22T01:02:07.001 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111676.log.gz 2024-09-22T01:02:07.001 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47327.log.gz 2024-09-22T01:02:07.001 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41249.log.gz 2024-09-22T01:02:07.001 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.98998.log.gz 2024-09-22T01:02:07.001 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108085.log.gz 2024-09-22T01:02:07.001 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75335.log.gz 2024-09-22T01:02:07.001 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74993.log.gz 2024-09-22T01:02:07.002 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78548.log.gz 2024-09-22T01:02:07.002 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55442.log.gz 2024-09-22T01:02:07.002 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.78799.log.gz 2024-09-22T01:02:07.002 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59986.log.gz 2024-09-22T01:02:07.002 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116129.log.gz 2024-09-22T01:02:07.002 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.87216.log.gz 2024-09-22T01:02:07.002 INFO:teuthology.orchestra.run.smithi060.stderr: 80.5% -- replaced with /var/log/ceph/cluster1-client.mirror.30991.log.gz 2024-09-22T01:02:07.002 INFO:teuthology.orchestra.run.smithi060.stderr: 80.5% -- replaced with /var/log/ceph/cluster1-client.mirror.109786.log.gz 2024-09-22T01:02:07.002 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78085.log.gz 2024-09-22T01:02:07.002 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89221.log.gz 2024-09-22T01:02:07.002 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.33277.log.gz 2024-09-22T01:02:07.002 INFO:teuthology.orchestra.run.smithi060.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32017.log.gz 2024-09-22T01:02:07.002 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66708.log.gz 2024-09-22T01:02:07.002 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.78443.log.gz 2024-09-22T01:02:07.002 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.96644.log.gz 2024-09-22T01:02:07.003 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.92975.log.gz 2024-09-22T01:02:07.003 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37986.log.gz 2024-09-22T01:02:07.003 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60614.log.gz 2024-09-22T01:02:07.003 INFO:teuthology.orchestra.run.smithi060.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.86107.log.gz 2024-09-22T01:02:07.003 INFO:teuthology.orchestra.run.smithi060.stderr: 80.5% -- replaced with /var/log/ceph/cluster1-client.mirror.31731.log.gz 2024-09-22T01:02:07.003 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.131819.log.gz 2024-09-22T01:02:07.003 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132535.log.gz 2024-09-22T01:02:07.003 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66667.log.gz 2024-09-22T01:02:07.003 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122755.log.gz 2024-09-22T01:02:07.003 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.105696.log.gz 2024-09-22T01:02:07.003 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.120539.log.gz 2024-09-22T01:02:07.003 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.87003.log.gz 2024-09-22T01:02:07.003 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.106774.log.gz 2024-09-22T01:02:07.003 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.75034.log.gz 2024-09-22T01:02:07.004 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75201.log.gz 2024-09-22T01:02:07.004 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114240.log.gz 2024-09-22T01:02:07.004 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.47537.log.gz 2024-09-22T01:02:07.004 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.76969.log.gz 2024-09-22T01:02:07.004 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.87763.log.gz 90.3% 2024-09-22T01:02:07.004 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.71286.log.gz 2024-09-22T01:02:07.004 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.96643.log.gz 2024-09-22T01:02:07.004 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.30687.log.gz 2024-09-22T01:02:07.004 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.96404.log.gz 2024-09-22T01:02:07.004 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.84791.log.gz 2024-09-22T01:02:07.004 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.96387.log.gz 2024-09-22T01:02:07.004 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.51398.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.95612.log.gz 2024-09-22T01:02:07.004 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.004 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.117453.log.gz 2024-09-22T01:02:07.004 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64779.log.gz 2024-09-22T01:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.95347.log.gz 2024-09-22T01:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.120035.log.gz 2024-09-22T01:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.129577.log.gz 2024-09-22T01:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.51673.log.gz 2024-09-22T01:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.120263.log.gz 2024-09-22T01:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.76859.log.gz 2024-09-22T01:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.126701.log.gz 2024-09-22T01:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78591.log.gz 2024-09-22T01:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.36975.log.gz 2024-09-22T01:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.129768.log.gz 2024-09-22T01:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.106118.log.gz 2024-09-22T01:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.50709.log.gz 2024-09-22T01:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69883.log.gz 2024-09-22T01:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.51396.log.gz 2024-09-22T01:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.140757.log.gz 2024-09-22T01:02:07.006 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.96920.log.gz 2024-09-22T01:02:07.006 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120316.log.gz 2024-09-22T01:02:07.006 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.95610.log.gz 2024-09-22T01:02:07.006 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112284.log.gz 2024-09-22T01:02:07.006 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.124140.log.gz 2024-09-22T01:02:07.006 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94309.log.gz 2024-09-22T01:02:07.006 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.104112.log.gz 2024-09-22T01:02:07.006 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42754.log.gz 2024-09-22T01:02:07.006 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60152.log.gz 2024-09-22T01:02:07.006 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111486.log.gz 2024-09-22T01:02:07.006 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67154.log.gz 2024-09-22T01:02:07.006 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135252.log.gz 2024-09-22T01:02:07.007 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72178.log.gz 2024-09-22T01:02:07.007 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91169.log.gz 2024-09-22T01:02:07.007 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.75910.log.gz 2024-09-22T01:02:07.007 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.133179.log.gz 2024-09-22T01:02:07.007 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63499.log.gz 2024-09-22T01:02:07.007 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38085.log.gz 2024-09-22T01:02:07.007 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66453.log.gz 2024-09-22T01:02:07.007 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56436.log.gz 2024-09-22T01:02:07.007 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.126241.log.gz 2024-09-22T01:02:07.007 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.83561.log.gz 2024-09-22T01:02:07.007 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% 90.6% -- replaced with /var/log/ceph/cluster1-client.admin.120062.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.87130.log.gz 2024-09-22T01:02:07.007 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.007 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35502.log.gz 2024-09-22T01:02:07.007 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.52445.log.gz 2024-09-22T01:02:07.007 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32113.log.gz 2024-09-22T01:02:07.007 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37478.log.gz 2024-09-22T01:02:07.008 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.89068.log.gz 2024-09-22T01:02:07.008 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60404.log.gz 2024-09-22T01:02:07.008 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.139880.log.gz 2024-09-22T01:02:07.008 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.39858.log.gz 2024-09-22T01:02:07.008 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61069.log.gz 2024-09-22T01:02:07.008 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53725.log.gz 2024-09-22T01:02:07.008 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71203.log.gz 2024-09-22T01:02:07.008 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40107.log.gz 2024-09-22T01:02:07.008 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84944.log.gz 2024-09-22T01:02:07.008 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71876.log.gz 2024-09-22T01:02:07.008 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41819.log.gz 2024-09-22T01:02:07.008 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108199.log.gz 2024-09-22T01:02:07.008 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82940.log.gz 2024-09-22T01:02:07.008 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41990.log.gz 2024-09-22T01:02:07.008 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.83106.log.gz 2024-09-22T01:02:07.009 INFO:teuthology.orchestra.run.smithi060.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.admin.83740.log.gz 2024-09-22T01:02:07.009 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134034.log.gz 2024-09-22T01:02:07.009 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.101148.log.gz 2024-09-22T01:02:07.009 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125652.log.gz 2024-09-22T01:02:07.009 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.64462.log.gz 2024-09-22T01:02:07.009 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.120609.log.gz 2024-09-22T01:02:07.009 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61058.log.gz 2024-09-22T01:02:07.009 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64716.log.gz 2024-09-22T01:02:07.009 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.39802.log.gz 2024-09-22T01:02:07.009 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63291.log.gz 2024-09-22T01:02:07.009 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.31792.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.34117.log.gz 2024-09-22T01:02:07.009 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.009 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.34158.log.gz 2024-09-22T01:02:07.009 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.132424.log.gz 2024-09-22T01:02:07.009 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.132572.log.gz 2024-09-22T01:02:07.010 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36259.log.gz 2024-09-22T01:02:07.010 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92470.log.gz 2024-09-22T01:02:07.010 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.76704.log.gz 2024-09-22T01:02:07.010 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74537.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.83621.log.gz 2024-09-22T01:02:07.010 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.010 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69816.log.gz 90.1% 2024-09-22T01:02:07.010 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.105458.log.gz 2024-09-22T01:02:07.010 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.118712.log.gz 2024-09-22T01:02:07.010 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55054.log.gz 2024-09-22T01:02:07.010 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79219.log.gz 2024-09-22T01:02:07.010 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38088.log.gz 2024-09-22T01:02:07.010 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67716.log.gz 2024-09-22T01:02:07.010 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113292.log.gz 2024-09-22T01:02:07.010 INFO:teuthology.orchestra.run.smithi060.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.admin.131145.log.gz 2024-09-22T01:02:07.011 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33488.log.gz 2024-09-22T01:02:07.011 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68683.log.gz 2024-09-22T01:02:07.011 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138196.log.gz 2024-09-22T01:02:07.011 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.133364.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.114451.log.gz 2024-09-22T01:02:07.011 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.011 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90580.log.gz 2024-09-22T01:02:07.011 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34451.log.gz 2024-09-22T01:02:07.011 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.46907.log.gz 2024-09-22T01:02:07.011 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55616.log.gz 2024-09-22T01:02:07.011 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125442.log.gz 2024-09-22T01:02:07.011 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125482.log.gz 2024-09-22T01:02:07.011 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68264.log.gz 2024-09-22T01:02:07.011 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.99965.log.gz 2024-09-22T01:02:07.011 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79896.log.gz 2024-09-22T01:02:07.012 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76037.log.gz 2024-09-22T01:02:07.012 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121809.log.gz 2024-09-22T01:02:07.012 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82036.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.89740.log.gz 2024-09-22T01:02:07.012 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.012 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89698.log.gz 2024-09-22T01:02:07.012 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114174.log.gz 2024-09-22T01:02:07.012 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37769.log.gz 2024-09-22T01:02:07.012 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34722.log.gz 2024-09-22T01:02:07.012 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104803.log.gz 2024-09-22T01:02:07.012 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49780.log.gz 2024-09-22T01:02:07.012 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125764.log.gz 2024-09-22T01:02:07.012 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.85044.log.gz 2024-09-22T01:02:07.012 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37985.log.gz 2024-09-22T01:02:07.012 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78758.log.gz 2024-09-22T01:02:07.012 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90664.log.gz 2024-09-22T01:02:07.013 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59712.log.gz 2024-09-22T01:02:07.013 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59562.log.gz 2024-09-22T01:02:07.013 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.78256.log.gz 2024-09-22T01:02:07.013 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.120692.log.gz 2024-09-22T01:02:07.013 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70109.log.gz 2024-09-22T01:02:07.013 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.127266.log.gz 89.8% 2024-09-22T01:02:07.013 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.80860.log.gz 2024-09-22T01:02:07.013 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65768.log.gz 2024-09-22T01:02:07.013 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.116903.log.gz 2024-09-22T01:02:07.013 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121451.log.gz 2024-09-22T01:02:07.013 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.140273.log.gz 2024-09-22T01:02:07.013 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90900.log.gz 2024-09-22T01:02:07.013 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136092.log.gz 2024-09-22T01:02:07.014 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.136133.log.gz 2024-09-22T01:02:07.014 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36112.log.gz 2024-09-22T01:02:07.014 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79263.log.gz 2024-09-22T01:02:07.014 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64990.log.gz 2024-09-22T01:02:07.014 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.94028.log.gz 2024-09-22T01:02:07.014 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88044.log.gz 2024-09-22T01:02:07.014 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.34871.log.gz 2024-09-22T01:02:07.014 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36113.log.gz 2024-09-22T01:02:07.014 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.89152.log.gz 2024-09-22T01:02:07.014 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137777.log.gz 2024-09-22T01:02:07.014 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79429.log.gz 2024-09-22T01:02:07.014 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113291.log.gz 2024-09-22T01:02:07.014 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79639.log.gz 2024-09-22T01:02:07.014 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133308.log.gz 2024-09-22T01:02:07.015 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79810.log.gz 2024-09-22T01:02:07.015 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113040.log.gz 2024-09-22T01:02:07.015 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125692.log.gz 2024-09-22T01:02:07.015 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124452.log.gz 2024-09-22T01:02:07.015 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102702.log.gz 2024-09-22T01:02:07.015 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40177.log.gz 2024-09-22T01:02:07.015 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.88615.log.gz 2024-09-22T01:02:07.015 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98703.log.gz 2024-09-22T01:02:07.015 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122438.log.gz 2024-09-22T01:02:07.015 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32414.log.gz 2024-09-22T01:02:07.015 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.34199.log.gz 2024-09-22T01:02:07.015 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65703.log.gz 2024-09-22T01:02:07.015 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91252.log.gz 2024-09-22T01:02:07.015 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102074.log.gz 2024-09-22T01:02:07.016 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34957.log.gz 2024-09-22T01:02:07.016 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49331.log.gz 2024-09-22T01:02:07.016 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33822.log.gz 2024-09-22T01:02:07.016 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.62268.log.gz 2024-09-22T01:02:07.016 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.88766.log.gz 2024-09-22T01:02:07.016 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90832.log.gz 2024-09-22T01:02:07.016 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60236.log.gz 2024-09-22T01:02:07.016 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.133309.log.gz 2024-09-22T01:02:07.016 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.35081.log.gz 2024-09-22T01:02:07.016 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54916.log.gz 2024-09-22T01:02:07.016 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.99510.log.gz 2024-09-22T01:02:07.016 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.32624.log.gz 2024-09-22T01:02:07.016 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113987.log.gz 2024-09-22T01:02:07.016 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114804.log.gz 2024-09-22T01:02:07.017 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49242.log.gz 2024-09-22T01:02:07.017 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71412.log.gz 2024-09-22T01:02:07.017 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115712.log.gz 2024-09-22T01:02:07.017 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.35922.log.gz 2024-09-22T01:02:07.017 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113754.log.gz 2024-09-22T01:02:07.017 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55532.log.gz 2024-09-22T01:02:07.017 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33612.log.gz 2024-09-22T01:02:07.017 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.32152.log.gz 2024-09-22T01:02:07.017 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.33821.log.gz 2024-09-22T01:02:07.017 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.31948.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.132091.log.gz 2024-09-22T01:02:07.017 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.017 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.85467.log.gz 2024-09-22T01:02:07.017 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.93629.log.gz 2024-09-22T01:02:07.017 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.41545.log.gz 2024-09-22T01:02:07.018 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% 90.2% -- replaced with /var/log/ceph/cluster2-client.admin.73489.log.gz 2024-09-22T01:02:07.018 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.67673.log.gz 2024-09-22T01:02:07.018 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101232.log.gz 2024-09-22T01:02:07.018 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123551.log.gz 2024-09-22T01:02:07.018 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104950.log.gz 2024-09-22T01:02:07.018 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.48085.log.gz 2024-09-22T01:02:07.018 INFO:teuthology.orchestra.run.smithi060.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-client.admin.131031.log.gz 2024-09-22T01:02:07.018 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.108153.log.gz 2024-09-22T01:02:07.018 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67153.log.gz 2024-09-22T01:02:07.018 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.111356.log.gz 2024-09-22T01:02:07.018 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37688.log.gz 2024-09-22T01:02:07.019 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34198.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.33780.log.gz 2024-09-22T01:02:07.019 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.019 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88943.log.gz 2024-09-22T01:02:07.019 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127792.log.gz 2024-09-22T01:02:07.019 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.62645.log.gz 2024-09-22T01:02:07.019 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86200.log.gz 2024-09-22T01:02:07.019 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104466.log.gz 2024-09-22T01:02:07.019 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.33462.log.gz 2024-09-22T01:02:07.019 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44046.log.gz 2024-09-22T01:02:07.019 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69523.log.gz 2024-09-22T01:02:07.019 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35248.log.gz 2024-09-22T01:02:07.019 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% 90.2% 96.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123593.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.80694.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.96940.log.gz 2024-09-22T01:02:07.019 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.019 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.019 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60572.log.gz 2024-09-22T01:02:07.020 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116550.log.gz 2024-09-22T01:02:07.020 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80525.log.gz 2024-09-22T01:02:07.020 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56245.log.gz 2024-09-22T01:02:07.020 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92370.log.gz 2024-09-22T01:02:07.020 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88177.log.gz 2024-09-22T01:02:07.020 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.129291.log.gz 2024-09-22T01:02:07.020 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103903.log.gz 2024-09-22T01:02:07.020 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57505.log.gz 2024-09-22T01:02:07.020 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138197.log.gz 2024-09-22T01:02:07.020 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36048.log.gz 2024-09-22T01:02:07.020 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.81996.log.gz 2024-09-22T01:02:07.020 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102072.log.gz 2024-09-22T01:02:07.020 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31643.log.gz 2024-09-22T01:02:07.020 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60532.log.gz 2024-09-22T01:02:07.020 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68178.log.gz 2024-09-22T01:02:07.020 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.139715.log.gz 2024-09-22T01:02:07.020 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90539.log.gz 2024-09-22T01:02:07.021 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104467.log.gz 2024-09-22T01:02:07.021 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92369.log.gz 2024-09-22T01:02:07.021 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79598.log.gz 2024-09-22T01:02:07.021 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45180.log.gz 2024-09-22T01:02:07.021 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.86314.log.gz 2024-09-22T01:02:07.021 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.126869.log.gz 2024-09-22T01:02:07.021 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.83984.log.gz 2024-09-22T01:02:07.021 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46210.log.gz 2024-09-22T01:02:07.021 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99968.log.gz 2024-09-22T01:02:07.021 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85724.log.gz 2024-09-22T01:02:07.021 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64906.log.gz 2024-09-22T01:02:07.021 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.140283.log.gz 2024-09-22T01:02:07.021 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.34242.log.gz 2024-09-22T01:02:07.021 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36763.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.129242.log.gz 2024-09-22T01:02:07.021 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.021 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132449.log.gz 2024-09-22T01:02:07.021 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43840.log.gz 2024-09-22T01:02:07.022 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52719.log.gz 2024-09-22T01:02:07.022 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.126535.log.gz 2024-09-22T01:02:07.022 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77101.log.gz 2024-09-22T01:02:07.022 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70427.log.gz 2024-09-22T01:02:07.022 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.32227.log.gz 2024-09-22T01:02:07.022 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.87870.log.gz 2024-09-22T01:02:07.022 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.40347.log.gz 2024-09-22T01:02:07.022 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.84400.log.gz 2024-09-22T01:02:07.022 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46991.log.gz 2024-09-22T01:02:07.022 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.34328.log.gz 2024-09-22T01:02:07.023 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.39415.log.gz 2024-09-22T01:02:07.023 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70908.log.gz 2024-09-22T01:02:07.023 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.133092.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.121046.log.gz 2024-09-22T01:02:07.023 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.023 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.75014.log.gz 2024-09-22T01:02:07.023 INFO:teuthology.orchestra.run.smithi060.stderr: 66.9% -- replaced with /var/log/ceph/cluster2-client.admin.30160.log.gz 2024-09-22T01:02:07.023 INFO:teuthology.orchestra.run.smithi060.stderr: 68.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108448.log.gz 2024-09-22T01:02:07.023 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.120357.log.gz 2024-09-22T01:02:07.024 INFO:teuthology.orchestra.run.smithi060.stderr: 80.5% -- replaced with /var/log/ceph/cluster1-client.mirror.87068.log.gz 2024-09-22T01:02:07.024 INFO:teuthology.orchestra.run.smithi060.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121157.log.gz 2024-09-22T01:02:07.024 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44592.log.gz 2024-09-22T01:02:07.024 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.78715.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.54617.log.gz 2024-09-22T01:02:07.024 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.024 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.140182.log.gz 2024-09-22T01:02:07.024 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.140173.log.gz 2024-09-22T01:02:07.024 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41799.log.gz 2024-09-22T01:02:07.024 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41477.log.gz 2024-09-22T01:02:07.024 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.120108.log.gz 2024-09-22T01:02:07.024 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% 91.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67128.log.gz 2024-09-22T01:02:07.024 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.92807.log.gz 2024-09-22T01:02:07.024 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.61953.log.gz 2024-09-22T01:02:07.024 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.129667.log.gz 2024-09-22T01:02:07.024 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124224.log.gz 2024-09-22T01:02:07.025 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52833.log.gz 2024-09-22T01:02:07.025 INFO:teuthology.orchestra.run.smithi060.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.117580.log.gz 2024-09-22T01:02:07.025 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.117220.log.gz 2024-09-22T01:02:07.025 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.106941.log.gz 2024-09-22T01:02:07.025 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.135713.log.gz 2024-09-22T01:02:07.025 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100016.log.gz 2024-09-22T01:02:07.025 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.117781.log.gz 2024-09-22T01:02:07.025 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109674.log.gz 2024-09-22T01:02:07.025 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34932.log.gz 2024-09-22T01:02:07.025 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45812.log.gz 2024-09-22T01:02:07.025 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137186.log.gz 2024-09-22T01:02:07.025 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38091.log.gz 91.8% 2024-09-22T01:02:07.025 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.118160.log.gz 2024-09-22T01:02:07.025 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.51754.log.gz 2024-09-22T01:02:07.025 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74651.log.gz 2024-09-22T01:02:07.025 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.119264.log.gz 2024-09-22T01:02:07.025 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.98190.log.gz 2024-09-22T01:02:07.026 INFO:teuthology.orchestra.run.smithi060.stderr: 68.3% -- replaced with /var/log/ceph/cluster1-client.mirror.85651.log.gz 2024-09-22T01:02:07.026 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.140811.log.gz 2024-09-22T01:02:07.026 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.117876.log.gz 2024-09-22T01:02:07.026 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.105140.log.gz 2024-09-22T01:02:07.026 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58303.log.gz 2024-09-22T01:02:07.026 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90690.log.gz 2024-09-22T01:02:07.026 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.82582.log.gz 2024-09-22T01:02:07.026 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36638.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.41431.log.gz 2024-09-22T01:02:07.026 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.026 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68557.log.gz 2024-09-22T01:02:07.026 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57759.log.gz 2024-09-22T01:02:07.026 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.33319.log.gz 2024-09-22T01:02:07.026 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.85041.log.gz 2024-09-22T01:02:07.026 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87067.log.gz 2024-09-22T01:02:07.026 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.106914.log.gz 2024-09-22T01:02:07.027 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65601.log.gz 2024-09-22T01:02:07.027 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34660.log.gz 2024-09-22T01:02:07.027 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112351.log.gz 2024-09-22T01:02:07.027 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.93919.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.88529.log.gz 2024-09-22T01:02:07.027 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.027 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76123.log.gz 2024-09-22T01:02:07.027 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70933.log.gz 2024-09-22T01:02:07.027 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90789.log.gz 2024-09-22T01:02:07.027 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60277.log.gz 2024-09-22T01:02:07.027 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.104111.log.gz 2024-09-22T01:02:07.027 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102851.log.gz 2024-09-22T01:02:07.027 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.36299.log.gz 2024-09-22T01:02:07.027 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104636.log.gz 2024-09-22T01:02:07.027 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32457.log.gz 2024-09-22T01:02:07.027 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49370.log.gz 2024-09-22T01:02:07.027 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.78844.log.gz 2024-09-22T01:02:07.027 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90579.log.gz 2024-09-22T01:02:07.028 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36510.log.gz 2024-09-22T01:02:07.028 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45897.log.gz 2024-09-22T01:02:07.028 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78484.log.gz 2024-09-22T01:02:07.028 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81238.log.gz 2024-09-22T01:02:07.028 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68413.log.gz 2024-09-22T01:02:07.028 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121155.log.gz 90.3% 2024-09-22T01:02:07.028 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.57381.log.gz 2024-09-22T01:02:07.028 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.96475.log.gz 2024-09-22T01:02:07.028 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.106377.log.gz 2024-09-22T01:02:07.028 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72114.log.gz 2024-09-22T01:02:07.028 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.83995.log.gz 2024-09-22T01:02:07.028 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37182.log.gz 2024-09-22T01:02:07.028 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.76374.log.gz 87.7% 2024-09-22T01:02:07.028 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.86766.log.gz 2024-09-22T01:02:07.029 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.36088.log.gz 2024-09-22T01:02:07.029 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.76498.log.gz 2024-09-22T01:02:07.029 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.117695.log.gz 2024-09-22T01:02:07.029 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37995.log.gz 90.7% 2024-09-22T01:02:07.029 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.35669.log.gz 2024-09-22T01:02:07.029 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.84226.log.gz 2024-09-22T01:02:07.029 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.118786.log.gz 2024-09-22T01:02:07.029 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.118015.log.gz 2024-09-22T01:02:07.029 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96577.log.gz 2024-09-22T01:02:07.029 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.129441.log.gz 2024-09-22T01:02:07.029 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71571.log.gz 2024-09-22T01:02:07.029 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.32371.log.gz 2024-09-22T01:02:07.029 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30864.log.gz 2024-09-22T01:02:07.030 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30729.log.gz 2024-09-22T01:02:07.030 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112349.log.gz 2024-09-22T01:02:07.030 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.40974.log.gz 2024-09-22T01:02:07.030 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43778.log.gz 2024-09-22T01:02:07.030 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78631.log.gz 2024-09-22T01:02:07.030 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35627.log.gz 2024-09-22T01:02:07.030 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70068.log.gz 2024-09-22T01:02:07.030 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101380.log.gz 2024-09-22T01:02:07.030 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45895.log.gz 2024-09-22T01:02:07.030 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90413.log.gz 2024-09-22T01:02:07.030 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.129578.log.gz 2024-09-22T01:02:07.030 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113334.log.gz 2024-09-22T01:02:07.030 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114616.log.gz 2024-09-22T01:02:07.030 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92167.log.gz 2024-09-22T01:02:07.030 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100961.log.gz 2024-09-22T01:02:07.030 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.112410.log.gz 2024-09-22T01:02:07.031 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136260.log.gz 90.1% 2024-09-22T01:02:07.031 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.35082.log.gz 2024-09-22T01:02:07.031 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100436.log.gz 2024-09-22T01:02:07.031 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.94938.log.gz 2024-09-22T01:02:07.031 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.104322.log.gz 2024-09-22T01:02:07.031 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.33739.log.gz 2024-09-22T01:02:07.031 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.95677.log.gz 2024-09-22T01:02:07.031 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.120924.log.gz 2024-09-22T01:02:07.031 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.48631.log.gz 2024-09-22T01:02:07.031 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.84493.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.139714.log.gz 2024-09-22T01:02:07.031 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.031 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.63473.log.gz 2024-09-22T01:02:07.031 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.37392.log.gz 91.7% 2024-09-22T01:02:07.031 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.74605.log.gz 2024-09-22T01:02:07.032 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.138028.log.gz 2024-09-22T01:02:07.032 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56332.log.gz 2024-09-22T01:02:07.032 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.63359.log.gz 2024-09-22T01:02:07.032 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47033.log.gz 2024-09-22T01:02:07.032 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36218.log.gz 2024-09-22T01:02:07.032 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.106672.log.gz 2024-09-22T01:02:07.032 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.84280.log.gz 2024-09-22T01:02:07.032 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.128980.log.gz 2024-09-22T01:02:07.032 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.118135.log.gz 2024-09-22T01:02:07.032 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.41089.log.gz 2024-09-22T01:02:07.032 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.118691.log.gz 2024-09-22T01:02:07.032 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.99532.log.gz 2024-09-22T01:02:07.032 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.107015.log.gz 2024-09-22T01:02:07.032 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.117585.log.gz 2024-09-22T01:02:07.032 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.124562.log.gz 2024-09-22T01:02:07.032 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.97624.log.gz 2024-09-22T01:02:07.032 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59143.log.gz 2024-09-22T01:02:07.033 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.82539.log.gz 2024-09-22T01:02:07.033 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.97122.log.gz 2024-09-22T01:02:07.033 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.admin.140803.log.gz 2024-09-22T01:02:07.033 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.117590.log.gz 2024-09-22T01:02:07.033 INFO:teuthology.orchestra.run.smithi060.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31834.log.gz 2024-09-22T01:02:07.033 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36219.log.gz 2024-09-22T01:02:07.033 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53289.log.gz 2024-09-22T01:02:07.033 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36111.log.gz 2024-09-22T01:02:07.033 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42199.log.gz 2024-09-22T01:02:07.033 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63747.log.gz 2024-09-22T01:02:07.033 INFO:teuthology.orchestra.run.smithi060.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.37243.log.gz 2024-09-22T01:02:07.033 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.37269.log.gz 2024-09-22T01:02:07.033 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.96894.log.gz 2024-09-22T01:02:07.033 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.62223.log.gz 2024-09-22T01:02:07.033 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68053.log.gz 2024-09-22T01:02:07.033 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.96479.log.gz 2024-09-22T01:02:07.034 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.admin.106385.log.gz 2024-09-22T01:02:07.034 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.72599.log.gz 2024-09-22T01:02:07.034 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.96325.log.gz 2024-09-22T01:02:07.034 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.95866.log.gz 2024-09-22T01:02:07.034 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.96018.log.gz 2024-09-22T01:02:07.034 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.119378.log.gz 2024-09-22T01:02:07.034 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63975.log.gz 2024-09-22T01:02:07.034 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.117221.log.gz 2024-09-22T01:02:07.034 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.119492.log.gz 2024-09-22T01:02:07.034 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.73744.log.gz 2024-09-22T01:02:07.034 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.129262.log.gz 2024-09-22T01:02:07.034 INFO:teuthology.orchestra.run.smithi060.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123092.log.gz 2024-09-22T01:02:07.034 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113821.log.gz 2024-09-22T01:02:07.034 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.61707.log.gz 2024-09-22T01:02:07.034 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65905.log.gz 2024-09-22T01:02:07.034 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70428.log.gz 2024-09-22T01:02:07.035 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.114031.log.gz 2024-09-22T01:02:07.035 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.47035.log.gz 2024-09-22T01:02:07.035 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.106366.log.gz 2024-09-22T01:02:07.035 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114552.log.gz 2024-09-22T01:02:07.035 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.39618.log.gz 2024-09-22T01:02:07.035 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109948.log.gz 2024-09-22T01:02:07.035 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31946.log.gz 2024-09-22T01:02:07.035 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.131723.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.50440.log.gz 2024-09-22T01:02:07.035 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.035 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114131.log.gz 2024-09-22T01:02:07.035 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90183.log.gz 2024-09-22T01:02:07.035 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58726.log.gz 2024-09-22T01:02:07.035 INFO:teuthology.orchestra.run.smithi060.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76121.log.gz 2024-09-22T01:02:07.035 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59186.log.gz 2024-09-22T01:02:07.036 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103481.log.gz 2024-09-22T01:02:07.036 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.104173.log.gz 2024-09-22T01:02:07.036 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57463.log.gz 90.0% 2024-09-22T01:02:07.036 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59482.log.gz 2024-09-22T01:02:07.036 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64499.log.gz 2024-09-22T01:02:07.036 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134664.log.gz 2024-09-22T01:02:07.036 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127900.log.gz 2024-09-22T01:02:07.036 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68474.log.gz 2024-09-22T01:02:07.036 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68893.log.gz 2024-09-22T01:02:07.036 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128172.log.gz 2024-09-22T01:02:07.036 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126533.log.gz 2024-09-22T01:02:07.036 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77144.log.gz 2024-09-22T01:02:07.036 INFO:teuthology.orchestra.run.smithi060.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115182.log.gz 2024-09-22T01:02:07.037 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80399.log.gz 2024-09-22T01:02:07.037 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65538.log.gz 2024-09-22T01:02:07.037 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51408.log.gz 2024-09-22T01:02:07.037 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32477.log.gz 2024-09-22T01:02:07.037 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57129.log.gz 2024-09-22T01:02:07.037 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.90749.log.gz 90.4% 2024-09-22T01:02:07.037 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.111931.log.gz 2024-09-22T01:02:07.037 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.120819.log.gz 2024-09-22T01:02:07.037 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59017.log.gz 2024-09-22T01:02:07.037 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.81156.log.gz 2024-09-22T01:02:07.037 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124014.log.gz 2024-09-22T01:02:07.037 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.36429.log.gz 2024-09-22T01:02:07.037 INFO:teuthology.orchestra.run.smithi060.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.97531.log.gz 2024-09-22T01:02:07.037 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.118389.log.gz 2024-09-22T01:02:07.037 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.34030.log.gz 2024-09-22T01:02:07.037 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.125586.log.gz 2024-09-22T01:02:07.038 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.78128.log.gz 2024-09-22T01:02:07.038 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113039.log.gz 2024-09-22T01:02:07.038 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.129676.log.gz 2024-09-22T01:02:07.038 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60235.log.gz 2024-09-22T01:02:07.038 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.73205.log.gz 2024-09-22T01:02:07.038 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62419.log.gz 2024-09-22T01:02:07.038 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.95799.log.gz 2024-09-22T01:02:07.038 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85858.log.gz 2024-09-22T01:02:07.038 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.97282.log.gz 2024-09-22T01:02:07.038 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.73642.log.gz 2024-09-22T01:02:07.038 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.106491.log.gz 2024-09-22T01:02:07.042 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.118867.log.gz 2024-09-22T01:02:07.042 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119424.log.gz 2024-09-22T01:02:07.042 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130507.log.gz 2024-09-22T01:02:07.042 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.140111.log.gz 2024-09-22T01:02:07.042 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119538.log.gz 2024-09-22T01:02:07.042 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.61587.log.gz 2024-09-22T01:02:07.042 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.50578.log.gz 2024-09-22T01:02:07.042 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63633.log.gz 2024-09-22T01:02:07.042 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64089.log.gz 2024-09-22T01:02:07.042 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60783.log.gz 2024-09-22T01:02:07.042 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.73655.log.gz 2024-09-22T01:02:07.042 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75107.log.gz 2024-09-22T01:02:07.042 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.96386.log.gz 2024-09-22T01:02:07.042 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.86086.log.gz 2024-09-22T01:02:07.043 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.86542.log.gz 2024-09-22T01:02:07.043 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97738.log.gz 2024-09-22T01:02:07.043 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97966.log.gz 2024-09-22T01:02:07.043 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.106369.log.gz 2024-09-22T01:02:07.043 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119766.log.gz 2024-09-22T01:02:07.043 INFO:teuthology.orchestra.run.smithi060.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.admin.51265.log.gz 2024-09-22T01:02:07.043 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.50949.log.gz 2024-09-22T01:02:07.043 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108313.log.gz 2024-09-22T01:02:07.043 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71245.log.gz 2024-09-22T01:02:07.043 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.128702.log.gz 2024-09-22T01:02:07.043 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.73038.log.gz 2024-09-22T01:02:07.043 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.73094.log.gz 2024-09-22T01:02:07.043 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.85630.log.gz 2024-09-22T01:02:07.043 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.admin.72990.log.gz 2024-09-22T01:02:07.043 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.39373.log.gz 2024-09-22T01:02:07.044 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41591.log.gz 2024-09-22T01:02:07.044 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60111.log.gz 2024-09-22T01:02:07.044 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.139921.log.gz 2024-09-22T01:02:07.044 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76331.log.gz 2024-09-22T01:02:07.044 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% 90.1% 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.77284.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.76818.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.58178.log.gz 2024-09-22T01:02:07.044 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.044 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:02:07.044 INFO:teuthology.orchestra.run.smithi060.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.75061.log.gz 2024-09-22T01:02:07.044 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.39213.log.gz 2024-09-22T01:02:07.044 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60151.log.gz 2024-09-22T01:02:07.044 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60362.log.gz 2024-09-22T01:02:07.045 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.129683.log.gz 2024-09-22T01:02:07.045 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130735.log.gz 2024-09-22T01:02:07.045 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.83958.log.gz 2024-09-22T01:02:07.051 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.84442.log.gz 2024-09-22T01:02:07.054 INFO:teuthology.orchestra.run.smithi060.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59985.log.gz 2024-09-22T01:02:07.055 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85972.log.gz 2024-09-22T01:02:07.060 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.74491.log.gz 2024-09-22T01:02:07.068 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.84021.log.gz 2024-09-22T01:02:07.069 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.84019.log.gz 2024-09-22T01:02:07.071 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.85584.log.gz 2024-09-22T01:02:07.073 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.96247.log.gz 2024-09-22T01:02:07.077 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.76628.log.gz 2024-09-22T01:02:07.084 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.131622.log.gz 2024-09-22T01:02:07.132 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.admin.51257.log.gz 2024-09-22T01:02:07.149 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.50569.log.gz 2024-09-22T01:02:07.158 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.50563.log.gz 2024-09-22T01:02:07.176 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.51596.log.gz 2024-09-22T01:02:07.188 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.101171.log.gz 2024-09-22T01:02:07.211 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.61615.log.gz 2024-09-22T01:02:07.219 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.62251.log.gz 2024-09-22T01:02:07.228 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59561.log.gz 2024-09-22T01:02:07.236 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.140447.log.gz 2024-09-22T01:02:07.245 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.39801.log.gz 2024-09-22T01:02:07.252 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70488.log.gz 2024-09-22T01:02:07.263 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.52217.log.gz 2024-09-22T01:02:07.263 INFO:teuthology.orchestra.run.smithi060.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103629.log.gz 2024-09-22T01:02:07.269 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.50564.log.gz 2024-09-22T01:02:07.389 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.73258.log.gz 2024-09-22T01:02:07.389 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63405.log.gz 2024-09-22T01:02:07.389 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60068.log.gz 2024-09-22T01:02:07.390 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60322.log.gz 2024-09-22T01:02:07.390 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63519.log.gz 2024-09-22T01:02:07.390 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63861.log.gz 2024-09-22T01:02:07.393 INFO:teuthology.orchestra.run.smithi060.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.mirror.104298.log.gz 2024-09-22T01:02:07.395 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.94926.log.gz 2024-09-22T01:02:07.401 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79305.log.gz 2024-09-22T01:02:07.402 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109111.log.gz 2024-09-22T01:02:07.404 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74765.log.gz 2024-09-22T01:02:07.407 INFO:teuthology.orchestra.run.smithi060.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.admin.95209.log.gz 2024-09-22T01:02:07.415 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.96215.log.gz 2024-09-22T01:02:07.418 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.95708.log.gz 2024-09-22T01:02:07.421 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58473.log.gz 2024-09-22T01:02:07.427 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.72830.log.gz 2024-09-22T01:02:07.428 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.73101.log.gz 2024-09-22T01:02:07.441 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112241.log.gz 2024-09-22T01:02:07.443 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75221.log.gz 2024-09-22T01:02:07.460 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.62269.log.gz 2024-09-22T01:02:07.474 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.106527.log.gz 2024-09-22T01:02:07.483 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.72598.log.gz 2024-09-22T01:02:07.500 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.39931.log.gz 2024-09-22T01:02:07.506 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60804.log.gz 2024-09-22T01:02:07.517 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.63245.log.gz 2024-09-22T01:02:07.532 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.admin.140119.log.gz 2024-09-22T01:02:07.532 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.140505.log.gz 2024-09-22T01:02:07.532 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.140537.log.gz 2024-09-22T01:02:07.533 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52263.log.gz 2024-09-22T01:02:07.534 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41705.log.gz 2024-09-22T01:02:07.535 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.117782.log.gz 2024-09-22T01:02:07.536 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.140275.log.gz 2024-09-22T01:02:07.537 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97054.log.gz 2024-09-22T01:02:07.540 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130393.log.gz 2024-09-22T01:02:07.545 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119880.log.gz 2024-09-22T01:02:07.547 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.108039.log.gz 2024-09-22T01:02:07.553 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131077.log.gz 2024-09-22T01:02:07.559 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74879.log.gz 2024-09-22T01:02:07.559 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.139723.log.gz 2024-09-22T01:02:07.562 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.118139.log.gz 2024-09-22T01:02:07.567 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.85513.log.gz 2024-09-22T01:02:07.570 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97396.log.gz 2024-09-22T01:02:07.571 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97510.log.gz 2024-09-22T01:02:07.572 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85744.log.gz 2024-09-22T01:02:07.572 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108541.log.gz 2024-09-22T01:02:07.574 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.84737.log.gz 2024-09-22T01:02:07.578 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.admin.128714.log.gz 2024-09-22T01:02:07.580 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.117570.log.gz 2024-09-22T01:02:07.582 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119994.log.gz 2024-09-22T01:02:07.584 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.admin.129022.log.gz 2024-09-22T01:02:07.592 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130621.log.gz 2024-09-22T01:02:07.598 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.admin.72583.log.gz 2024-09-22T01:02:07.603 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119652.log.gz 2024-09-22T01:02:07.606 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86428.log.gz 2024-09-22T01:02:07.609 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75449.log.gz 2024-09-22T01:02:07.611 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108883.log.gz 2024-09-22T01:02:07.631 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.39154.log.gz 2024-09-22T01:02:07.642 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.39407.log.gz 2024-09-22T01:02:07.643 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130963.log.gz 2024-09-22T01:02:07.644 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.62245.log.gz 2024-09-22T01:02:07.654 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108655.log.gz 2024-09-22T01:02:07.658 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.50764.log.gz 2024-09-22T01:02:07.663 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.74377.log.gz 2024-09-22T01:02:07.670 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76701.log.gz 2024-09-22T01:02:07.694 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131191.log.gz 2024-09-22T01:02:07.697 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.61698.log.gz 2024-09-22T01:02:07.701 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64203.log.gz 2024-09-22T01:02:07.717 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74423.log.gz 2024-09-22T01:02:07.736 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.78883.log.gz 2024-09-22T01:02:07.773 INFO:teuthology.orchestra.run.smithi060.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.50570.log.gz 2024-09-22T01:02:07.872 INFO:teuthology.orchestra.run.smithi060.stderr: 93.9% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-09-22T01:02:07.875 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41135.log.gz 2024-09-22T01:02:07.877 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.51371.log.gz 2024-09-22T01:02:08.122 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41020.log.gz 2024-09-22T01:02:08.169 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41363.log.gz 2024-09-22T01:02:09.044 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53175.log.gz 2024-09-22T01:02:09.048 INFO:teuthology.orchestra.run.smithi060.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.51739.log.gz 2024-09-22T01:02:09.065 INFO:teuthology.orchestra.run.smithi060.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53061.log.gz 2024-09-22T01:02:09.258 INFO:teuthology.orchestra.run.smithi060.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52605.log.gz 2024-09-22T01:02:10.200 INFO:teuthology.orchestra.run.smithi031.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-09-22T01:02:35.168 INFO:teuthology.orchestra.run.smithi060.stderr: 96.1% -- replaced with /var/log/ceph/cluster2-client.mirror.30306.log.gz 2024-09-22T01:03:01.592 INFO:teuthology.orchestra.run.smithi060.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-09-22T01:03:48.392 INFO:teuthology.orchestra.run.smithi060.stderr: 94.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.30306.log.gz 2024-09-22T01:08:57.636 INFO:teuthology.orchestra.run.smithi060.stderr: 97.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30335.log.gz 2024-09-22T01:11:38.034 INFO:teuthology.orchestra.run.smithi060.stderr: 94.5% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-09-22T01:12:26.761 INFO:teuthology.orchestra.run.smithi060.stderr: 94.6% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-09-22T01:12:44.894 INFO:teuthology.orchestra.run.smithi060.stderr: 94.5% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-09-22T01:12:53.547 INFO:teuthology.orchestra.run.smithi060.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.30335.log.gz 2024-09-22T01:12:53.548 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T01:12:53.548 INFO:teuthology.orchestra.run.smithi060.stderr:real 11m0.785s 2024-09-22T01:12:53.548 INFO:teuthology.orchestra.run.smithi060.stderr:user 10m23.741s 2024-09-22T01:12:53.548 INFO:teuthology.orchestra.run.smithi060.stderr:sys 0m34.819s 2024-09-22T01:14:11.732 INFO:teuthology.orchestra.run.smithi031.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-09-22T01:14:47.582 INFO:teuthology.orchestra.run.smithi031.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-09-22T01:15:09.081 INFO:teuthology.orchestra.run.smithi031.stderr: 94.1% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-09-22T01:15:09.082 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-22T01:15:09.082 INFO:teuthology.orchestra.run.smithi031.stderr:real 13m16.345s 2024-09-22T01:15:09.082 INFO:teuthology.orchestra.run.smithi031.stderr:user 13m43.745s 2024-09-22T01:15:09.082 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m41.233s 2024-09-22T01:15:09.083 INFO:tasks.ceph:Archiving logs... 2024-09-22T01:15:09.084 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914544/remote/smithi031/log 2024-09-22T01:15:09.085 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-22T01:15:41.974 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914544/remote/smithi060/log 2024-09-22T01:15:41.975 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-22T01:16:08.104 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-22T01:16:08.141 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-22T01:16:08.141 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-22T01:16:08.144 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-22T01:16:08.240 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-22T01:16:08.251 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-22T01:16:08.251 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T01:16:08.254 DEBUG:teuthology.orchestra.run.smithi060:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T01:16:08.332 INFO:teuthology.orchestra.run.smithi031.stdout: remote refid st t when poll reach delay offset jitter 2024-09-22T01:16:08.332 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================== 2024-09-22T01:16:08.332 INFO:teuthology.orchestra.run.smithi031.stdout:+hv01.front.sepi 67.205.162.81 3 u 251 256 377 0.188 -3.197 0.578 2024-09-22T01:16:08.332 INFO:teuthology.orchestra.run.smithi031.stdout:*hv02.front.sepi 63.231.80.2 3 u 150 256 377 0.151 -2.538 0.331 2024-09-22T01:16:08.332 INFO:teuthology.orchestra.run.smithi031.stdout:+hv03.front.sepi 74.6.168.72 3 u 146 256 377 0.177 -4.429 0.371 2024-09-22T01:16:08.332 INFO:teuthology.orchestra.run.smithi031.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-22T01:16:08.344 INFO:teuthology.orchestra.run.smithi060.stdout: remote refid st t when poll reach delay offset jitter 2024-09-22T01:16:08.344 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================== 2024-09-22T01:16:08.344 INFO:teuthology.orchestra.run.smithi060.stdout:*hv01.front.sepi 67.205.162.81 3 u 129 256 377 0.112 -0.997 0.214 2024-09-22T01:16:08.344 INFO:teuthology.orchestra.run.smithi060.stdout:+hv02.front.sepi 63.231.80.2 3 u 199 256 377 0.146 +0.384 0.274 2024-09-22T01:16:08.344 INFO:teuthology.orchestra.run.smithi060.stdout:+hv03.front.sepi 74.6.168.72 3 u 158 256 377 0.168 -1.467 0.156 2024-09-22T01:16:08.344 INFO:teuthology.orchestra.run.smithi060.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-22T01:16:08.345 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-22T01:16:08.355 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-22T01:16:08.355 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-22T01:16:08.363 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-22T01:16:08.392 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-22T01:16:08.404 INFO:teuthology.task.internal:Duration was 5867.068336 seconds 2024-09-22T01:16:08.404 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-22T01:16:08.413 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-22T01:16:08.413 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-22T01:16:08.416 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-22T01:16:08.682 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-22T01:16:08.682 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2024-09-22T01:16:08.682 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-22T01:16:08.692 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi060.front.sepia.ceph.com 2024-09-22T01:16:08.693 DEBUG:teuthology.orchestra.run.smithi060:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-22T01:16:08.747 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-22T01:16:08.747 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-22T01:16:08.749 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-22T01:16:08.854 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-22T01:16:08.855 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-22T01:16:08.857 DEBUG:teuthology.orchestra.run.smithi060:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-22T01:16:09.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-22T01:16:09.515 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-22T01:16:09.515 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-22T01:16:09.545 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-22T01:16:09.608 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-22T01:16:09.618 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-22T01:16:09.620 DEBUG:teuthology.orchestra.run.smithi060:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-22T01:16:09.643 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core 2024-09-22T01:16:09.663 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2024-09-22T01:16:09.685 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-22T01:16:09.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T01:16:09.689 DEBUG:teuthology.orchestra.run.smithi060:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-22T01:16:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T01:16:09.724 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-22T01:16:09.734 INFO:teuthology.task.internal:Transferring archived files... 2024-09-22T01:16:09.734 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914544/remote/smithi031 2024-09-22T01:16:09.735 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-22T01:16:09.755 DEBUG:teuthology.misc:Transferring archived files from smithi060:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914544/remote/smithi060 2024-09-22T01:16:09.755 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-22T01:16:09.789 INFO:teuthology.task.internal:Removing archive directory... 2024-09-22T01:16:09.789 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-22T01:16:09.795 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-22T01:16:09.833 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-22T01:16:09.844 INFO:teuthology.task.internal:Not uploading archives. 2024-09-22T01:16:09.844 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-22T01:16:09.854 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-22T01:16:09.854 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-22T01:16:09.856 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-22T01:16:09.860 INFO:teuthology.orchestra.run.smithi031.stdout: 658641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 22 01:16 /home/ubuntu/cephtest 2024-09-22T01:16:09.881 INFO:teuthology.orchestra.run.smithi060.stdout: 658641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 22 01:16 /home/ubuntu/cephtest 2024-09-22T01:16:09.883 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-22T01:16:09.893 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-22T01:16:09.916 INFO:teuthology.run:Summary data: description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/compare-mirror-images-krbd} duration: 5867.068336248398 failure_reason: '"2024-09-21T23:51:22.607862+0000 mon.a (mon.0) 248 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY)" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-09-22T01:16:09.916 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T01:16:09.997 INFO:teuthology.run:FAIL